From 7864ed98a9f207fd116c44b45ccf672f26def490 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 13:29:31 -0400 Subject: [PATCH 01/19] Add a working plan for a medium-scale testnet simulation. Set current focus to a reproducible ~100-user campaign on real seed statements, with local validation and gated remote stages. --- .../TESTNET-SIMULATION-PLAN.md | 198 ++++++++++++++++++ focus.md | 4 +- 2 files changed, 201 insertions(+), 1 deletion(-) create mode 100644 fake-data-generation/TESTNET-SIMULATION-PLAN.md diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md new file mode 100644 index 00000000..6862fb90 --- /dev/null +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -0,0 +1,198 @@ +# Medium-scale realistic testnet simulation — working plan + +Tell a fresh LLM: **read this file, then do the next unchecked item under [Next](#next).** Keep [`PLAN.md`](./PLAN.md)'s four data jobs separate: this work combines curated Job C statements with disposable Job D actors, but does not turn fake activity into seed content. Deployment and shared-lab recovery still belong to [`../workflow/testnet-working-plan.md`](../workflow/testnet-working-plan.md). + +## Goal + +Build a reproducible, observable campaign that makes a testnet deployment look and behave like a small but real Commonality community: + +- about **100 synthetic users** with persistent, disposable wallets; +- **8–12 coherent causes** built from real, human-accepted seed statements; +- initially **30–50 statements**, **15–30 projects**, and roughly **1,000–3,000 successful actions**; +- uneven participation, overlapping memberships, different roles and engagement levels, and a small controlled amount of disagreement and inactivity; +- a report that reconciles intended actions, mined transactions, indexed events, derived state, and representative UI pages. + +This is a **product-realism and operational-confidence test**, not a maximum-throughput benchmark. Success means the populated system is credible and inspectable, not merely that 100 wallets can submit transactions. + +## Why this is worth doing + +Tiny and demo seeds show that individual stories can work. A medium-sized world can reveal a different class of failures before real people encounter them: + +- cause boards, discovery, rankings, and trust-filtered views that become confusing at non-toy density; +- indexing omissions, duplicates, lag, replay problems, and client-fold performance; +- incoherent simulation behavior that makes apparently healthy product surfaces meaningless; +- operational limits around wallet provisioning, gas, payment tokens, RPC quotas, IPFS publication, resumability, and diagnosis; +- whether a founder can look at a populated vertical and see something compelling enough to demonstrate. + +## Boundaries + +### Real content, synthetic activity + +- Seed statements must come from accepted files under [`seed-content/`](./seed-content/) or pass the existing [`statement-generation.md`](./statement-generation.md) acceptance process first. +- Prefer useful public-goods planks and a small number of already-accepted bridge clusters. Do not mass-generate political triples for this campaign. +- Synthetic users, projects, transactions, and relationships must be identifiable as test data. They are not examples to present as genuine adoption. +- Do not put campaign-only statements into `universe.json`, tiny seed, or accepted seed content merely to make the workload convenient. + +### Not a browser-bot campaign + +Most volume should use contract/SDK calls. A smaller browser sample verifies that the resulting state is visible and understandable through the actual product. One hundred browser sessions would test a different thing and make failures harder to localize. + +### Not a testnet recovery mechanism + +The shared two-person lab must be healthy before it receives the campaign. Do not use fake activity to cover empty official graphs, broken services, or incomplete first-user journeys. If the readiness gate below fails, continue locally. + +### Not yet a scalability claim + +The campaign may expose scale problems, but 100 users and a few thousand actions cannot justify claims about production capacity or millions of users. A later load-test design can reuse the harness with a different workload and acceptance criteria. + +## Target world + +The campaign manifest should specify the world independently of execution mechanics. A good first distribution is: + +| Element | Target | Shape | +|---|---:|---| +| Users | 100 | Stable generated identities; clearly labelled campaign wallets | +| Causes | 8–12 | 2–3 large, 3–4 medium, several small or nearly inactive | +| Statements | 30–50 | Accepted planks plus a limited number of meaningful close/medium variants and bridge statements | +| Membership | 1–3 causes/user | Correlated with persona; overlapping rather than partitioned camps | +| Projects | 15–30 | Plausible projects aligned to actual planks; mix of active, weak, successful, and unfunded | +| Activity | 1,000–3,000 successful writes | Spread across phases/time; not one RPC burst | +| Roles | several of each | Cause founders, project founders, supporters, delegates, attesters/mediators, lurkers, power users | + +Use deliberately uneven participation. Uniformly assigning the same number of users and actions to every cause produces clean-looking but uninformative data. + +The first corpus should lean toward ordinary public goods—open-source maintenance, local food, accessibility, science/open data, literacy, and civic infrastructure—plus a small sample of the accepted abortion, immigration, crime, and LGBT-schools bridge clusters. Political content should not dominate the apparent product. + +## Campaign artifacts + +Each campaign needs a stable ID and an artifact directory that contains no committed secrets: + +- input manifest version and deterministic random seed; +- selected statement IDs/CIDs and their accepted-source fingerprints; +- persona and cause-membership assignments; +- generated wallet addresses, with private keys stored separately and gitignored; +- planned action graph, including prerequisites between actions; +- submitted transaction hashes and receipts; +- per-action status: planned, submitted, mined, indexed, verified, skipped, or failed; +- gas/payment-token funding ledger and actual consumption; +- indexer reconciliation results and lag measurements; +- representative cause/user/project URLs and browser observations; +- final summary with failure categories and enough detail to reproduce the run. + +The manifest must make reruns deterministic without requiring private keys to be checked into Git. + +## Execution design + +### Separate planning from execution + +Generate a complete campaign plan before sending transactions. Validate references, prerequisites, balances, estimated gas, expected write count, and contract addresses in a dry run. Execution should consume that plan rather than making consequential random choices while running. + +### Explicit environments + +Do not point the current `gen:large` command at Base Sepolia. Add an explicit remote campaign mode with: + +- verified chain ID and deployed address manifest; +- no deployment, Hardhat keys, free minting, or localhost assumptions; +- bounded concurrency, rate limiting, retries with backoff, and configurable pacing; +- idempotency/resume behavior based on recorded transaction and indexed state; +- a global transaction and native-token budget that aborts closed when exceeded; +- an unmistakable confirmation/opt-in flag for mutating a remote chain. + +Keep the fast local simulator useful. Extract or share behavior/model code where practical instead of turning every local seed path into remote-aware code. + +### Persona-driven behavior + +Actions should follow declared roles and cause membership: + +- users believe or disbelieve statements that make sense for their persona; +- projects align to statements that describe their actual outcome; +- supporters fund projects in their causes with a non-uniform distribution; +- delegation follows a generated trust graph and has valid predecessor notes; +- implication attestations use accepted/evaluated relationships, never a random truth value; +- some users lapse, change a belief, revoke a delegation, or remain mostly inactive; +- a few deliberately invalid actions may test rejection paths, but they are a separate labelled workload and do not pollute success metrics. + +### Provisioning + +Before execution, calculate rather than guess: + +- native gas required per wallet and for central publishing/provisioning accounts; +- payment-token needs by persona and action plan; +- whether sponsored gas is intentionally part of this test; +- faucet/RPC/provider limits and total expected wall-clock time. + +Fund wallets minimally and in phases. Never reuse the local Hardhat default keys on testnet. If a freely mintable test payment token is used, verify the deployed token's permissions instead of relying on a failed transfer followed by a blind `mintTo` fallback. + +## Verification and success criteria + +### Reconciliation + +For every action type used in the campaign, compare: + +1. the action the manifest intended; +2. transaction submission and receipt status; +3. the corresponding raw indexed event/entity; +4. SDK-derived state or aggregate; +5. UI visibility where the action is user-facing. + +The report must distinguish transaction failures, expected contract reverts, RPC/provider failures, indexer lag, permanent indexing omissions, fold errors, and UI presentation defects. + +### Required metrics + +- planned/submitted/mined/indexed/verified counts by action type; +- failure and retry counts by classified cause; +- gas used and native/payment-token cost by action type and campaign phase; +- receipt latency and transaction throughput; +- time-to-index distribution and chain-head lag during/after each phase; +- cause size, statement support, project funding, delegation, and trust-graph distributions; +- duplicate/missing entities and mismatched derived totals; +- sampled browser journey results and links to inspected pages. + +### Completion gate + +This focus is complete when: + +- a clean local campaign with the target world is deterministic and passes reconciliation; +- a 10-user remote canary can be safely resumed and fully reconciles; +- staged 25-user and 100-user testnet runs complete within explicit budgets; +- no unexplained mined-but-unindexed actions remain after the agreed settling window; +- representative UI inspection finds the world intelligible, not merely populated; +- the final report identifies bottlenecks and product findings without claiming production-scale capacity; +- rerun, cleanup/retention, and synthetic-data identification procedures are documented. + +## Testnet readiness gate + +Do not begin a mutating remote phase until all of these are true: + +- [`../workflow/testnet-working-plan.md`](../workflow/testnet-working-plan.md)'s shared-lab milestone is boring enough that new failures can be attributed to the campaign; +- the relevant read-only verifier leaves pass immediately before the run; +- the official implication/trust path needed by the chosen statements is intentionally populated and working; +- the remote runner verifies chain ID, contract bytecode/addresses, indexer endpoint, and campaign budget; +- wallet secrets have a storage and deletion/retention procedure; +- Adam has approved the particular campaign manifest, budget, pacing, and testnet window. + +After each remote phase, stop and reconcile before increasing the user or action count. A failed gate sends the work back to local diagnosis; it does not justify pushing more traffic. + +## Next + +Do these in order unless Adam names a different item. Keep each item small enough to review independently. + +1. **[ ] Freeze the v1 campaign schema and world.** Inventory reusable simulator and seed-content types; choose the initial 8–12 causes and 30–50 accepted statements; define personas, roles, distributions, action prerequisites, artifact layout, synthetic-data labels, and deterministic-seed behavior. This is design plus schema/tests, not testnet mutation. +2. **[ ] Build a deterministic local campaign planner.** Emit the complete planned world/action graph and a cost/write estimate without private keys or transactions. Add validation for impossible actions, missing statement references, unsuitable implications, and distribution targets. +3. **[ ] Split local assumptions from reusable execution.** Introduce explicit chain configuration and adapters for deployment/provisioning. Keep existing tiny/demo/local commands working. Remote mode must refuse Hardhat keys, wrong chain IDs, missing bytecode, and implicit deployment/minting. +4. **[ ] Add resumable, budgeted execution.** Persist action state and transaction hashes; add bounded concurrency, pacing, retry classification, stop/resume, transaction caps, and native-token budget enforcement. Prove interruption and resume locally. +5. **[ ] Make simulated behavior persona- and cause-aware.** Replace random implication truth and mechanically unrelated selections in the campaign path. Generate plausible project, belief, funding, alignment, and delegation histories from the manifest. +6. **[ ] Build transaction-to-indexer deep reconciliation.** Cover every selected action type, derived totals, duplicates, and settling-window behavior. Produce machine-readable results plus a concise human report. +7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. +8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. +9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. +10. **[ ] Run staged 25-user then 100-user campaigns.** Re-run health/preflight and reconcile after each phase. Do not automatically advance after a failed or materially surprising phase. +11. **[ ] Publish the campaign report and follow-up decisions.** Record product findings, operational limits, indexing correctness/latency, gas/provider costs, and the explicitly unsupported scalability claims. Move concrete fixes to the appropriate backlog or a new focus; do not let this plan become a permanent catch-all. + +## Relationship to existing tools + +- [`runSimulation.ts`](./runSimulation.ts) remains the starting implementation reference, not a safe testnet command. +- [`PLAN.md`](./PLAN.md) remains the source of truth for the four fake/seed-data jobs and statement-volume sequencing. +- [`statement-generation.md`](./statement-generation.md) remains the only acceptance pipeline for new real statements. +- [`../workflow/testnet-working-plan.md`](../workflow/testnet-working-plan.md) owns shared-lab health and deployment operations. +- The verifier should gain campaign-specific checks only when their desired inputs, cadence, and autonomy tier are understood; the first implementation can produce standalone reconciliation artifacts. diff --git a/focus.md b/focus.md index 41e86a32..6e1d7b25 100644 --- a/focus.md +++ b/focus.md @@ -2,4 +2,6 @@ High-level work we are actually concentrating on right now. Not a to-do list — those live in [TODO.md](./TODO.md), [causestarter/TODO.md](./causestarter/TODO.md), and the [testnet working plan](./workflow/testnet-working-plan.md). At most three items. Keep this file current. -Nothing is currently listed. The last closed item was [third-party project proposals and alignment](./specs/product/third-party-project-proposals-and-alignment.md). +- **Medium-scale realistic testnet simulation.** Build a reproducible, observable campaign with about 100 synthetic users acting across a useful corpus of real seed statements, validate it locally, then run it on testnet through gated 10/25/100-user stages. See the [working plan](./fake-data-generation/TESTNET-SIMULATION-PLAN.md). + +The last closed item was [third-party project proposals and alignment](./specs/product/third-party-project-proposals-and-alignment.md). From 6d29d9255f32aa908e3feaeab7d342dba6f9e365 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 13:33:45 -0400 Subject: [PATCH 02/19] Define v1 realistic campaign schema --- .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignSchema.ts | 144 ++++++++++++++++++ .../campaigns/medium-realistic-v1.json | 118 ++++++++++++++ .../test/campaignSchema.test.ts | 52 +++++++ 4 files changed, 315 insertions(+), 1 deletion(-) create mode 100644 fake-data-generation/campaignSchema.ts create mode 100644 fake-data-generation/campaigns/medium-realistic-v1.json create mode 100644 fake-data-generation/test/campaignSchema.test.ts diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 6862fb90..b03429f2 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -177,7 +177,7 @@ After each remote phase, stop and reconcile before increasing the user or action Do these in order unless Adam names a different item. Keep each item small enough to review independently. -1. **[ ] Freeze the v1 campaign schema and world.** Inventory reusable simulator and seed-content types; choose the initial 8–12 causes and 30–50 accepted statements; define personas, roles, distributions, action prerequisites, artifact layout, synthetic-data labels, and deterministic-seed behavior. This is design plus schema/tests, not testnet mutation. +1. **[x] Freeze the v1 campaign schema and world.** [`campaignSchema.ts`](./campaignSchema.ts) defines and validates the versioned contract; [`campaigns/medium-realistic-v1.json`](./campaigns/medium-realistic-v1.json) selects 10 causes and 46 accepted statements and fixes the 100-user persona mix, uneven activity, action prerequisites/targets, artifact layout, synthetic-data label, and deterministic PRNG seed. Tests resolve every statement reference against accepted seed content and enforce the target shape. This remains planning-only and contains no wallets or transactions. 2. **[ ] Build a deterministic local campaign planner.** Emit the complete planned world/action graph and a cost/write estimate without private keys or transactions. Add validation for impossible actions, missing statement references, unsuitable implications, and distribution targets. 3. **[ ] Split local assumptions from reusable execution.** Introduce explicit chain configuration and adapters for deployment/provisioning. Keep existing tiny/demo/local commands working. Remote mode must refuse Hardhat keys, wrong chain IDs, missing bytecode, and implicit deployment/minting. 4. **[ ] Add resumable, budgeted execution.** Persist action state and transaction hashes; add bounded concurrency, pacing, retry classification, stop/resume, transaction caps, and native-token budget enforcement. Prove interruption and resume locally. diff --git a/fake-data-generation/campaignSchema.ts b/fake-data-generation/campaignSchema.ts new file mode 100644 index 00000000..d5d5ec6a --- /dev/null +++ b/fake-data-generation/campaignSchema.ts @@ -0,0 +1,144 @@ +export const CAMPAIGN_SCHEMA_VERSION = 'commonality-testnet-campaign-v1' as const; + +export type CampaignRole = + | 'cause-founder' + | 'project-founder' + | 'supporter' + | 'delegate' + | 'attester' + | 'mediator' + | 'lurker' + | 'power-user'; + +export type CampaignActionType = + | 'publish-statement' + | 'create-cause' + | 'set-belief' + | 'attest-implication' + | 'create-project' + | 'attest-alignment' + | 'fund-project' + | 'deposit-note' + | 'delegate-note' + | 'revoke-delegation'; + +export interface CampaignStatementRef { + collectionId: string; + groupId: string; + statementId: string; +} + +export interface CampaignCause { + id: string; + title: string; + activityTier: 'large' | 'medium' | 'small'; + membershipWeight: number; + statementRefs: CampaignStatementRef[]; +} + +export interface CampaignPersona { + id: string; + count: number; + roles: CampaignRole[]; + causesPerUser: { min: number; max: number }; + activityWeight: number; + inactivityRate: number; + fundingWeight: number; +} + +export interface CampaignActionRule { + type: CampaignActionType; + prerequisites: CampaignActionType[]; + targetCount: { min: number; max: number }; +} + +export interface CampaignManifestV1 { + schema: typeof CAMPAIGN_SCHEMA_VERSION; + campaign: { + id: string; + label: string; + syntheticDataLabel: string; + userCount: number; + deterministicSeed: string; + randomAlgorithm: 'xoshiro128**'; + }; + sourcePolicy: { + kind: 'accepted-seed-content-only'; + fingerprintAlgorithm: 'sha256'; + excludeCollections: string[]; + }; + causes: CampaignCause[]; + personas: CampaignPersona[]; + actionRules: CampaignActionRule[]; + artifactLayout: Record; +} + +function requireUnique(values: string[], label: string): void { + const duplicates = values.filter((value, index) => values.indexOf(value) !== index); + if (duplicates.length > 0) throw new Error(`${label} contains duplicates: ${[...new Set(duplicates)].join(', ')}`); +} + +function validateCampaignIdentity(manifest: CampaignManifestV1): void { + const campaign = manifest.campaign; + if (!/^[a-z0-9][a-z0-9-]+$/.test(campaign.id)) throw new Error('campaign.id must be a stable kebab-case ID'); + if (!campaign.syntheticDataLabel.toLowerCase().includes('synthetic')) { + throw new Error('campaign.syntheticDataLabel must unmistakably identify synthetic data'); + } + if (!campaign.deterministicSeed) throw new Error('campaign.deterministicSeed is required'); +} + +function validateCauses(manifest: CampaignManifestV1): void { + if (manifest.causes.length < 8 || manifest.causes.length > 12) throw new Error('campaign must contain 8-12 causes'); + requireUnique(manifest.causes.map((cause) => cause.id), 'cause IDs'); + const statementRefs = manifest.causes.flatMap((cause) => cause.statementRefs); + const statementKeys = statementRefs.map((ref) => `${ref.collectionId}/${ref.groupId}/${ref.statementId}`); + requireUnique(statementKeys, 'statement references'); + if (statementRefs.length < 30 || statementRefs.length > 50) throw new Error('campaign must contain 30-50 statements'); + if (manifest.causes.some((cause) => cause.membershipWeight <= 0 || cause.statementRefs.length === 0)) { + throw new Error('every cause needs a positive membership weight and at least one statement'); + } +} + +function validatePersonas(manifest: CampaignManifestV1): void { + requireUnique(manifest.personas.map((persona) => persona.id), 'persona IDs'); + const personaCount = manifest.personas.reduce((total, persona) => total + persona.count, 0); + if (personaCount !== manifest.campaign.userCount) throw new Error(`persona counts total ${personaCount}, expected ${manifest.campaign.userCount}`); + for (const persona of manifest.personas) { + if (persona.count <= 0 || persona.activityWeight < 0 || persona.fundingWeight < 0) throw new Error(`invalid weights/count for persona ${persona.id}`); + if (persona.causesPerUser.min < 1 || persona.causesPerUser.max > 3 || persona.causesPerUser.min > persona.causesPerUser.max) { + throw new Error(`persona ${persona.id} must join 1-3 causes`); + } + if (persona.inactivityRate < 0 || persona.inactivityRate > 1) throw new Error(`invalid inactivityRate for persona ${persona.id}`); + } +} + +function validateActionRules(manifest: CampaignManifestV1): void { + requireUnique(manifest.actionRules.map((rule) => rule.type), 'action rule types'); + const actionTypes = new Set(manifest.actionRules.map((rule) => rule.type)); + for (const rule of manifest.actionRules) { + if (rule.targetCount.min < 0 || rule.targetCount.min > rule.targetCount.max) throw new Error(`invalid target count for ${rule.type}`); + for (const prerequisite of rule.prerequisites) { + if (!actionTypes.has(prerequisite)) throw new Error(`${rule.type} has unknown prerequisite ${prerequisite}`); + if (prerequisite === rule.type) throw new Error(`${rule.type} cannot depend on itself`); + } + } +} + +function validateArtifactLayout(manifest: CampaignManifestV1): void { + const requiredArtifacts = ['manifest', 'statementCatalog', 'assignments', 'walletAddresses', 'walletSecrets', 'actionPlan', 'executionState', 'fundingLedger', 'reconciliation', 'browserObservations', 'summary']; + for (const artifact of requiredArtifacts) { + if (!manifest.artifactLayout[artifact]) throw new Error(`artifactLayout.${artifact} is required`); + } + if (!manifest.artifactLayout.walletSecrets.startsWith('../secrets/')) { + throw new Error('wallet secrets must live outside the campaign artifact directory'); + } +} + +export function validateCampaignManifest(manifest: CampaignManifestV1): void { + if (manifest.schema !== CAMPAIGN_SCHEMA_VERSION) throw new Error(`unsupported campaign schema: ${String(manifest.schema)}`); + validateCampaignIdentity(manifest); + validateCauses(manifest); + validatePersonas(manifest); + validateActionRules(manifest); + validateArtifactLayout(manifest); +} diff --git a/fake-data-generation/campaigns/medium-realistic-v1.json b/fake-data-generation/campaigns/medium-realistic-v1.json new file mode 100644 index 00000000..072d2ccb --- /dev/null +++ b/fake-data-generation/campaigns/medium-realistic-v1.json @@ -0,0 +1,118 @@ +{ + "schema": "commonality-testnet-campaign-v1", + "campaign": { + "id": "medium-realistic-v1", + "label": "Medium realistic campaign v1", + "syntheticDataLabel": "SYNTHETIC TESTNET CAMPAIGN — NOT REAL USERS OR ADOPTION", + "userCount": 100, + "deterministicSeed": "commonality-medium-realistic-v1-2026-09-11", + "randomAlgorithm": "xoshiro128**" + }, + "sourcePolicy": { + "kind": "accepted-seed-content-only", + "fingerprintAlgorithm": "sha256", + "excludeCollections": ["proliferation"] + }, + "causes": [ + { "id": "open-source", "title": "Open-source public infrastructure", "activityTier": "large", "membershipWeight": 16, "statementRefs": [ + { "collectionId": "fundable-projects", "groupId": "open-source-and-public-infrastructure", "statementId": "open-source-public-infrastructure" }, + { "collectionId": "fundable-projects", "groupId": "open-source-and-public-infrastructure", "statementId": "fund-critical-maintainers" }, + { "collectionId": "simple-causes", "groupId": "open-source-maintenance", "statementId": "oss-libraries-kept-up" }, + { "collectionId": "simple-causes", "groupId": "open-source-maintenance", "statementId": "linux-kept-up" }, + { "collectionId": "simple-causes", "groupId": "open-source-maintenance", "statementId": "oss-not-single-vendor-capture" } + ]}, + { "id": "local-food", "title": "Local food systems", "activityTier": "large", "membershipWeight": 15, "statementRefs": [ + { "collectionId": "simple-causes", "groupId": "local-food-planks", "statementId": "neighborhood-growing" }, + { "collectionId": "simple-causes", "groupId": "local-food-planks", "statementId": "farmers-markets-direct-connect" }, + { "collectionId": "simple-causes", "groupId": "local-food-planks", "statementId": "community-supported-agriculture" }, + { "collectionId": "simple-causes", "groupId": "local-food-planks", "statementId": "working-local-farms" }, + { "collectionId": "simple-causes", "groupId": "local-food-planks", "statementId": "shorter-food-supply-chains" } + ]}, + { "id": "open-science", "title": "Open and trustworthy science", "activityTier": "large", "membershipWeight": 14, "statementRefs": [ + { "collectionId": "fundable-projects", "groupId": "scientific-research", "statementId": "open-access-scientific-publishing" }, + { "collectionId": "fundable-projects", "groupId": "scientific-research", "statementId": "research-neglected-diseases" }, + { "collectionId": "fundable-projects", "groupId": "scientific-research", "statementId": "independent-replication-studies" }, + { "collectionId": "fundable-projects", "groupId": "scientific-research", "statementId": "conflict-free-scientific-research" } + ]}, + { "id": "education", "title": "Education and literacy", "activityTier": "medium", "membershipWeight": 12, "statementRefs": [ + { "collectionId": "fundable-projects", "groupId": "education", "statementId": "vocational-training" }, + { "collectionId": "fundable-projects", "groupId": "education", "statementId": "educational-support-for-disadvantaged-kids" }, + { "collectionId": "fundable-projects", "groupId": "education", "statementId": "learning-outcomes-research" }, + { "collectionId": "fundable-projects", "groupId": "education", "statementId": "classic-literature-and-history" } + ]}, + { "id": "environment", "title": "Environmental resilience", "activityTier": "medium", "membershipWeight": 11, "statementRefs": [ + { "collectionId": "fundable-projects", "groupId": "environment-and-sustainability", "statementId": "clean-energy-research" }, + { "collectionId": "fundable-projects", "groupId": "environment-and-sustainability", "statementId": "independent-environmental-monitoring" }, + { "collectionId": "fundable-projects", "groupId": "environment-and-sustainability", "statementId": "conservation-of-local-natural-areas" }, + { "collectionId": "fundable-projects", "groupId": "environment-and-sustainability", "statementId": "sustainable-agriculture-research" } + ]}, + { "id": "digital-rights", "title": "Digital rights and civic infrastructure", "activityTier": "medium", "membershipWeight": 10, "statementRefs": [ + { "collectionId": "fundable-projects", "groupId": "civil-liberties", "statementId": "censorship-resistant-publishing" }, + { "collectionId": "fundable-projects", "groupId": "civil-liberties", "statementId": "protect-from-surveillance" }, + { "collectionId": "fundable-projects", "groupId": "civil-liberties", "statementId": "privacy-tools-for-ordinary-people" }, + { "collectionId": "fundable-projects", "groupId": "civil-liberties", "statementId": "decentralized-social-media-alternatives" } + ]}, + { "id": "abortion-common-ground", "title": "Abortion common ground", "activityTier": "small", "membershipWeight": 7, "statementRefs": [ + { "collectionId": "compromise-abortion", "groupId": "abortion-gestational-cutoff", "statementId": "natural-left" }, + { "collectionId": "compromise-abortion", "groupId": "abortion-gestational-cutoff", "statementId": "natural-right" }, + { "collectionId": "compromise-abortion", "groupId": "abortion-gestational-cutoff", "statementId": "modified-left" }, + { "collectionId": "compromise-abortion", "groupId": "abortion-gestational-cutoff", "statementId": "modified-right" }, + { "collectionId": "compromise-abortion", "groupId": "abortion-gestational-cutoff", "statementId": "commonality" } + ]}, + { "id": "immigration-common-ground", "title": "Immigration common ground", "activityTier": "small", "membershipWeight": 6, "statementRefs": [ + { "collectionId": "compromise-immigration", "groupId": "immigration-criminal-priority-overlap", "statementId": "natural-left" }, + { "collectionId": "compromise-immigration", "groupId": "immigration-criminal-priority-overlap", "statementId": "natural-right" }, + { "collectionId": "compromise-immigration", "groupId": "immigration-criminal-priority-overlap", "statementId": "modified-left" }, + { "collectionId": "compromise-immigration", "groupId": "immigration-criminal-priority-overlap", "statementId": "modified-right" }, + { "collectionId": "compromise-immigration", "groupId": "immigration-criminal-priority-overlap", "statementId": "commonality" } + ]}, + { "id": "violent-crime-common-ground", "title": "Violent crime common ground", "activityTier": "small", "membershipWeight": 5, "statementRefs": [ + { "collectionId": "crime-repeat-offenders", "groupId": "crime-repeat-offender-concentration", "statementId": "natural-left" }, + { "collectionId": "crime-repeat-offenders", "groupId": "crime-repeat-offender-concentration", "statementId": "natural-right" }, + { "collectionId": "crime-repeat-offenders", "groupId": "crime-repeat-offender-concentration", "statementId": "modified-left" }, + { "collectionId": "crime-repeat-offenders", "groupId": "crime-repeat-offender-concentration", "statementId": "modified-right" }, + { "collectionId": "crime-repeat-offenders", "groupId": "crime-repeat-offender-concentration", "statementId": "commonality" } + ]}, + { "id": "schools-common-ground", "title": "Schools and LGBT common ground", "activityTier": "small", "membershipWeight": 4, "statementRefs": [ + { "collectionId": "lgbt-schools", "groupId": "lgbt-schools-pushing", "statementId": "natural-left" }, + { "collectionId": "lgbt-schools", "groupId": "lgbt-schools-pushing", "statementId": "natural-right" }, + { "collectionId": "lgbt-schools", "groupId": "lgbt-schools-pushing", "statementId": "modified-left" }, + { "collectionId": "lgbt-schools", "groupId": "lgbt-schools-pushing", "statementId": "modified-right" }, + { "collectionId": "lgbt-schools", "groupId": "lgbt-schools-pushing", "statementId": "commonality" } + ]} + ], + "personas": [ + { "id": "quiet-neighbor", "count": 24, "roles": ["lurker", "supporter"], "causesPerUser": { "min": 1, "max": 2 }, "activityWeight": 0.25, "inactivityRate": 0.35, "fundingWeight": 0.5 }, + { "id": "regular-supporter", "count": 30, "roles": ["supporter"], "causesPerUser": { "min": 1, "max": 3 }, "activityWeight": 1, "inactivityRate": 0.08, "fundingWeight": 1 }, + { "id": "community-organizer", "count": 14, "roles": ["cause-founder", "supporter"], "causesPerUser": { "min": 2, "max": 3 }, "activityWeight": 2, "inactivityRate": 0, "fundingWeight": 1.2 }, + { "id": "project-builder", "count": 14, "roles": ["project-founder", "supporter"], "causesPerUser": { "min": 1, "max": 2 }, "activityWeight": 1.7, "inactivityRate": 0.03, "fundingWeight": 0.8 }, + { "id": "trusted-delegate", "count": 8, "roles": ["delegate", "supporter"], "causesPerUser": { "min": 2, "max": 3 }, "activityWeight": 2.2, "inactivityRate": 0, "fundingWeight": 1.4 }, + { "id": "bridge-mediator", "count": 6, "roles": ["attester", "mediator"], "causesPerUser": { "min": 2, "max": 3 }, "activityWeight": 2.4, "inactivityRate": 0, "fundingWeight": 0.6 }, + { "id": "power-user", "count": 4, "roles": ["power-user", "supporter", "delegate"], "causesPerUser": { "min": 3, "max": 3 }, "activityWeight": 4, "inactivityRate": 0, "fundingWeight": 2.5 } + ], + "actionRules": [ + { "type": "publish-statement", "prerequisites": [], "targetCount": { "min": 46, "max": 46 } }, + { "type": "create-cause", "prerequisites": ["publish-statement"], "targetCount": { "min": 10, "max": 10 } }, + { "type": "set-belief", "prerequisites": ["publish-statement"], "targetCount": { "min": 350, "max": 650 } }, + { "type": "attest-implication", "prerequisites": ["publish-statement"], "targetCount": { "min": 20, "max": 50 } }, + { "type": "create-project", "prerequisites": ["create-cause"], "targetCount": { "min": 18, "max": 26 } }, + { "type": "attest-alignment", "prerequisites": ["create-project", "publish-statement"], "targetCount": { "min": 30, "max": 65 } }, + { "type": "fund-project", "prerequisites": ["create-project", "attest-alignment"], "targetCount": { "min": 350, "max": 900 } }, + { "type": "deposit-note", "prerequisites": [], "targetCount": { "min": 120, "max": 260 } }, + { "type": "delegate-note", "prerequisites": ["deposit-note"], "targetCount": { "min": 80, "max": 200 } }, + { "type": "revoke-delegation", "prerequisites": ["delegate-note"], "targetCount": { "min": 10, "max": 35 } } + ], + "artifactLayout": { + "manifest": "manifest.json", + "statementCatalog": "plan/statements.json", + "assignments": "plan/assignments.json", + "walletAddresses": "plan/wallet-addresses.json", + "walletSecrets": "../secrets/medium-realistic-v1.wallets.json", + "actionPlan": "plan/actions.json", + "executionState": "execution/actions.jsonl", + "fundingLedger": "execution/funding-ledger.json", + "reconciliation": "reports/reconciliation.json", + "browserObservations": "reports/browser-observations.json", + "summary": "reports/summary.md" + } +} diff --git a/fake-data-generation/test/campaignSchema.test.ts b/fake-data-generation/test/campaignSchema.test.ts new file mode 100644 index 00000000..04ff85b7 --- /dev/null +++ b/fake-data-generation/test/campaignSchema.test.ts @@ -0,0 +1,52 @@ +import assert from 'node:assert/strict'; +import { readFile } from 'node:fs/promises'; +import test from 'node:test'; +import { CAMPAIGN_SCHEMA_VERSION, type CampaignManifestV1, validateCampaignManifest } from '../campaignSchema.js'; +import { flattenSeedStatements, loadSeedCollections } from '../seed-content-format.js'; + +async function loadManifest(): Promise { + return JSON.parse(await readFile(new URL('../campaigns/medium-realistic-v1.json', import.meta.url), 'utf8')) as CampaignManifestV1; +} + +test('v1 medium campaign satisfies the frozen schema and target shape', async () => { + const manifest = await loadManifest(); + validateCampaignManifest(manifest); + + assert.equal(manifest.schema, CAMPAIGN_SCHEMA_VERSION); + assert.equal(manifest.campaign.userCount, 100); + assert.equal(manifest.causes.length, 10); + assert.equal(manifest.causes.flatMap((cause) => cause.statementRefs).length, 46); + assert.deepEqual( + Object.fromEntries(['large', 'medium', 'small'].map((tier) => [tier, manifest.causes.filter((cause) => cause.activityTier === tier).length])), + { large: 3, medium: 3, small: 4 }, + ); +}); + +test('every campaign statement resolves to accepted, non-proliferated seed content', async () => { + const manifest = await loadManifest(); + const records = flattenSeedStatements(await loadSeedCollections()); + const available = new Set(records.map(({ collection, group, statement }) => `${collection.id}/${group.id}/${statement.id}`)); + + for (const cause of manifest.causes) { + for (const ref of cause.statementRefs) { + assert.ok(!manifest.sourcePolicy.excludeCollections.includes(ref.collectionId)); + assert.ok(available.has(`${ref.collectionId}/${ref.groupId}/${ref.statementId}`), `missing statement ${JSON.stringify(ref)}`); + } + } +}); + +test('v1 action targets describe roughly 1,000-3,000 successful writes', async () => { + const manifest = await loadManifest(); + const minimum = manifest.actionRules.reduce((total, rule) => total + rule.targetCount.min, 0); + const maximum = manifest.actionRules.reduce((total, rule) => total + rule.targetCount.max, 0); + + assert.ok(minimum >= 1_000, `minimum action target was ${minimum}`); + assert.ok(maximum <= 3_000, `maximum action target was ${maximum}`); +}); + +test('manifest validation rejects secrets inside campaign artifacts', async () => { + const manifest = await loadManifest(); + const invalid = structuredClone(manifest); + invalid.artifactLayout.walletSecrets = 'plan/private-keys.json'; + assert.throws(() => validateCampaignManifest(invalid), /outside the campaign artifact directory/); +}); From 2504edf68ec8ac436aa01a2a80fa811aad9b6081 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 13:40:06 -0400 Subject: [PATCH 03/19] Add deterministic campaign planner --- fake-data-generation/README.md | 2 + .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignPlanner.ts | 259 ++++++++++++++++++ fake-data-generation/package.json | 1 + .../test/campaignPlanner.test.ts | 43 +++ 5 files changed, 306 insertions(+), 1 deletion(-) create mode 100644 fake-data-generation/campaignPlanner.ts create mode 100644 fake-data-generation/test/campaignPlanner.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index e9bacb14..976509f4 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -11,6 +11,8 @@ This directory is **four jobs**, not one “generate fake data” switch: The rest of this README is the **generative simulation** (jobs 1, 2, 4): randomly generated users and on-chain actions. Job 3 is the JSON + attester loop, not `runSimulation`. +The medium realistic campaign has a separate planning-only command: `npm run gen:campaign:plan`. It deterministically writes the selected statement catalog, unprovisioned wallet slots, persona/cause/project assignments, complete prerequisite-linked action graph, and gas/payment-token estimate under `output/campaigns/medium-realistic-v1/`. A later provisioning step binds public addresses to those slots; planning neither creates private keys nor connects to a chain. + ## Overview The generative testing suite consists of: diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index b03429f2..af671891 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -178,7 +178,7 @@ After each remote phase, stop and reconcile before increasing the user or action Do these in order unless Adam names a different item. Keep each item small enough to review independently. 1. **[x] Freeze the v1 campaign schema and world.** [`campaignSchema.ts`](./campaignSchema.ts) defines and validates the versioned contract; [`campaigns/medium-realistic-v1.json`](./campaigns/medium-realistic-v1.json) selects 10 causes and 46 accepted statements and fixes the 100-user persona mix, uneven activity, action prerequisites/targets, artifact layout, synthetic-data label, and deterministic PRNG seed. Tests resolve every statement reference against accepted seed content and enforce the target shape. This remains planning-only and contains no wallets or transactions. -2. **[ ] Build a deterministic local campaign planner.** Emit the complete planned world/action graph and a cost/write estimate without private keys or transactions. Add validation for impossible actions, missing statement references, unsuitable implications, and distribution targets. +2. **[x] Build a deterministic local campaign planner.** [`campaignPlanner.ts`](./campaignPlanner.ts) expands the manifest into stable statement fingerprints, 100 persona/cause assignments, unprovisioned wallet slots, projects, and a prerequisite-linked action graph before any chain connection. `npm run gen:campaign:plan` writes the planning artifacts plus per-action write/gas and payment-token estimates. Tests cover deterministic replay, distribution bounds, accepted bridge-role implications, and impossible dependencies; validation also rejects missing/excluded statements and out-of-range action counts. 3. **[ ] Split local assumptions from reusable execution.** Introduce explicit chain configuration and adapters for deployment/provisioning. Keep existing tiny/demo/local commands working. Remote mode must refuse Hardhat keys, wrong chain IDs, missing bytecode, and implicit deployment/minting. 4. **[ ] Add resumable, budgeted execution.** Persist action state and transaction hashes; add bounded concurrency, pacing, retry classification, stop/resume, transaction caps, and native-token budget enforcement. Prove interruption and resume locally. 5. **[ ] Make simulated behavior persona- and cause-aware.** Replace random implication truth and mechanically unrelated selections in the campaign path. Generate plausible project, belief, funding, alignment, and delegation histories from the manifest. diff --git a/fake-data-generation/campaignPlanner.ts b/fake-data-generation/campaignPlanner.ts new file mode 100644 index 00000000..2f37a9a4 --- /dev/null +++ b/fake-data-generation/campaignPlanner.ts @@ -0,0 +1,259 @@ +import { createHash } from 'node:crypto'; +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { type CampaignActionType, type CampaignManifestV1, type CampaignRole, validateCampaignManifest } from './campaignSchema.js'; +import { flattenSeedStatements, loadSeedCollections } from './seed-content-format.js'; + +export const CAMPAIGN_PLAN_VERSION = 'commonality-campaign-plan-v1' as const; + +type JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; + +export interface PlannedStatement { + id: string; + causeId: string; + source: { collectionId: string; groupId: string; statementId: string; fingerprint: string }; + text: string; + role: string | null; +} + +export interface PlannedUser { + id: string; + walletSlot: string; + personaId: string; + roles: CampaignRole[]; + causeIds: string[]; + inactive: boolean; + activityWeight: number; + fundingWeight: number; +} + +export interface PlannedProject { + id: string; + causeId: string; + founderUserId: string; + statementIds: string[]; +} + +export interface PlannedAction { + id: string; + sequence: number; + type: CampaignActionType; + actorUserId: string | null; + causeId?: string; + statementId?: string; + projectId?: string; + noteId?: string; + delegateUserId?: string; + implication?: { fromStatementId: string; toStatementId: string; evidence: 'accepted-bridge-role-pair' }; + dependsOn: string[]; +} + +export interface CampaignPlan { + version: typeof CAMPAIGN_PLAN_VERSION; + campaignId: string; + deterministicSeed: string; + manifestFingerprint: string; + statements: PlannedStatement[]; + users: PlannedUser[]; + projects: PlannedProject[]; + actions: PlannedAction[]; + estimate: { + writesByType: Record; + totalWrites: number; + estimatedGasByType: Record; + estimatedTotalGas: number; + assumptions: { gasUnitsPerWrite: Record; paymentTokenUnitsPerFunding: number }; + estimatedPaymentTokenUnits: number; + }; +} + +const GAS_UNITS: Record = { + 'publish-statement': 180_000, 'create-cause': 120_000, 'set-belief': 90_000, + 'attest-implication': 130_000, 'create-project': 1_100_000, 'attest-alignment': 130_000, + 'fund-project': 180_000, 'deposit-note': 150_000, 'delegate-note': 100_000, 'revoke-delegation': 90_000, +}; + +function stableJson(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(stableJson).join(',')}]`; + if (value && typeof value === 'object') { + return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(',')}}`; + } + return JSON.stringify(value); +} + +function sha256(value: string): string { return createHash('sha256').update(value).digest('hex'); } + +class Xoshiro128StarStar { + private readonly state: Uint32Array; + constructor(seed: string) { + const bytes = createHash('sha256').update(seed).digest(); + this.state = new Uint32Array(4); + for (let index = 0; index < 4; index++) this.state[index] = bytes.readUInt32LE(index * 4); + } + next(): number { + const s = this.state; + const result = Math.imul(((Math.imul(s[1], 5) << 7) | (Math.imul(s[1], 5) >>> 25)) >>> 0, 9) >>> 0; + const temporary = (s[1] << 9) >>> 0; + s[2] ^= s[0]; s[3] ^= s[1]; s[1] ^= s[2]; s[0] ^= s[3]; s[2] ^= temporary; + s[3] = ((s[3] << 11) | (s[3] >>> 21)) >>> 0; + return result / 0x1_0000_0000; + } + integer(min: number, max: number): number { return min + Math.floor(this.next() * (max - min + 1)); } + pick(items: readonly T[]): T { if (items.length === 0) throw new Error('cannot choose from an empty list'); return items[Math.floor(this.next() * items.length)]; } + weighted(items: readonly T[], weight: (item: T) => number): T { + const total = items.reduce((sum, item) => sum + weight(item), 0); + if (total <= 0) throw new Error('weighted choice needs a positive total weight'); + let cursor = this.next() * total; + for (const item of items) { cursor -= weight(item); if (cursor < 0) return item; } + return items[items.length - 1]; + } + shuffle(items: readonly T[]): T[] { + const result = [...items]; + for (let index = result.length - 1; index > 0; index--) { const other = this.integer(0, index); [result[index], result[other]] = [result[other], result[index]]; } + return result; + } +} + +function statementKey(ref: { collectionId: string; groupId: string; statementId: string }): string { + return `${ref.collectionId}/${ref.groupId}/${ref.statementId}`; +} + +function allocateCount(rule: CampaignManifestV1['actionRules'][number], random: Xoshiro128StarStar): number { + return random.integer(rule.targetCount.min, rule.targetCount.max); +} + +function chooseDistinctCauses(manifest: CampaignManifestV1, count: number, random: Xoshiro128StarStar): string[] { + const remaining = [...manifest.causes]; + const chosen: string[] = []; + while (chosen.length < count) { + const cause = random.weighted(remaining, (item) => item.membershipWeight); + chosen.push(cause.id); + remaining.splice(remaining.indexOf(cause), 1); + } + return chosen; +} + +function validateImplicationPair(from: PlannedStatement, to: PlannedStatement): void { + const sameGroup = from.source.collectionId === to.source.collectionId && from.source.groupId === to.source.groupId; + const approved = (from.role === 'modified-left' || from.role === 'modified-right') && to.role === 'commonality'; + if (!sameGroup || !approved) throw new Error(`unsuitable implication ${from.id} -> ${to.id}: pair lacks accepted bridge-role evidence`); +} + +export async function buildCampaignPlan(manifest: CampaignManifestV1): Promise { + validateCampaignManifest(manifest); + const random = new Xoshiro128StarStar(manifest.campaign.deterministicSeed); + const records = flattenSeedStatements(await loadSeedCollections()); + const recordByKey = new Map(records.map((record) => [statementKey({ collectionId: record.collection.id, groupId: record.group.id, statementId: record.statement.id }), record])); + const statements: PlannedStatement[] = manifest.causes.flatMap((cause) => cause.statementRefs.map((ref) => { + const record = recordByKey.get(statementKey(ref)); + if (!record || manifest.sourcePolicy.excludeCollections.includes(ref.collectionId)) throw new Error(`missing or excluded statement reference ${statementKey(ref)}`); + return { id: `statement-${cause.id}-${ref.statementId}`, causeId: cause.id, source: { ...ref, fingerprint: sha256(stableJson(record.statement)) }, text: record.statement.text, role: record.statement.role ?? null }; + })); + + const users: PlannedUser[] = []; + for (const persona of manifest.personas) for (let index = 0; index < persona.count; index++) { + const id = `user-${String(users.length + 1).padStart(3, '0')}`; + const causeCount = random.integer(persona.causesPerUser.min, persona.causesPerUser.max); + users.push({ id, walletSlot: `wallet-${id}`, personaId: persona.id, roles: persona.roles, causeIds: chooseDistinctCauses(manifest, causeCount, random), inactive: random.next() < persona.inactivityRate, activityWeight: persona.activityWeight, fundingWeight: persona.fundingWeight }); + } + const activeUsers = users.filter((user) => !user.inactive && user.activityWeight > 0); + if (activeUsers.length === 0) throw new Error('campaign has no active users'); + + const countByType = Object.fromEntries(manifest.actionRules.map((rule) => [rule.type, allocateCount(rule, random)])) as Record; + countByType['publish-statement'] = statements.length; + countByType['create-cause'] = manifest.causes.length; + const projectFounders = activeUsers.filter((user) => user.roles.includes('project-founder') || user.roles.includes('power-user')); + const attesters = activeUsers.filter((user) => user.roles.includes('attester')); + const delegates = activeUsers.filter((user) => user.roles.includes('delegate')); + if (projectFounders.length === 0 || attesters.length === 0 || delegates.length === 0) throw new Error('campaign lacks an active project founder, attester, or delegate'); + + const projects: PlannedProject[] = Array.from({ length: countByType['create-project'] }, (_, index) => { + const founder = random.pick(projectFounders); + const causeId = random.pick(founder.causeIds); + const candidates = statements.filter((statement) => statement.causeId === causeId); + return { id: `project-${String(index + 1).padStart(3, '0')}`, causeId, founderUserId: founder.id, statementIds: random.shuffle(candidates).slice(0, random.integer(1, Math.min(3, candidates.length))).map((statement) => statement.id) }; + }); + + const actions: PlannedAction[] = []; + const add = (action: Omit): PlannedAction => { const value = { ...action, id: `action-${String(actions.length + 1).padStart(5, '0')}`, sequence: actions.length + 1 }; actions.push(value); return value; }; + const publishes = new Map(statements.map((statement) => [statement.id, add({ type: 'publish-statement', actorUserId: null, causeId: statement.causeId, statementId: statement.id, dependsOn: [] })])); + const causes = new Map(manifest.causes.map((cause) => [cause.id, add({ type: 'create-cause', actorUserId: random.pick(activeUsers.filter((user) => user.roles.includes('cause-founder') || user.roles.includes('power-user'))).id, causeId: cause.id, dependsOn: cause.statementRefs.map((ref) => publishes.get(statements.find((statement) => statementKey(statement.source) === statementKey(ref))!.id)!.id) })])); + + for (let index = 0; index < countByType['set-belief']; index++) { + const actor = random.weighted(activeUsers, (user) => user.activityWeight); + const statement = random.pick(statements.filter((item) => actor.causeIds.includes(item.causeId))); + add({ type: 'set-belief', actorUserId: actor.id, causeId: statement.causeId, statementId: statement.id, dependsOn: [publishes.get(statement.id)!.id] }); + } + const implicationPairs = manifest.causes.flatMap((cause) => { + const groupStatements = statements.filter((statement) => statement.causeId === cause.id); + const commonality = groupStatements.find((statement) => statement.role === 'commonality'); + return commonality ? groupStatements.filter((statement) => statement.role === 'modified-left' || statement.role === 'modified-right').map((from) => ({ from, to: commonality })) : []; + }); + if (countByType['attest-implication'] > 0 && implicationPairs.length === 0) throw new Error('no suitable accepted implication pairs are available'); + for (let index = 0; index < countByType['attest-implication']; index++) { + const pair = implicationPairs[index % implicationPairs.length]; validateImplicationPair(pair.from, pair.to); + add({ type: 'attest-implication', actorUserId: random.pick(attesters).id, causeId: pair.from.causeId, implication: { fromStatementId: pair.from.id, toStatementId: pair.to.id, evidence: 'accepted-bridge-role-pair' }, dependsOn: [publishes.get(pair.from.id)!.id, publishes.get(pair.to.id)!.id] }); + } + const createProjects = new Map(projects.map((project) => [project.id, add({ type: 'create-project', actorUserId: project.founderUserId, causeId: project.causeId, projectId: project.id, dependsOn: [causes.get(project.causeId)!.id] })])); + const alignments: PlannedAction[] = []; + for (let index = 0; index < countByType['attest-alignment']; index++) { + const project = projects[index % projects.length]; const statementId = project.statementIds[index % project.statementIds.length]; + alignments.push(add({ type: 'attest-alignment', actorUserId: random.pick(attesters).id, causeId: project.causeId, projectId: project.id, statementId, dependsOn: [createProjects.get(project.id)!.id, publishes.get(statementId)!.id] })); + } + for (let index = 0; index < countByType['fund-project']; index++) { + const project = random.pick(projects); const supporters = activeUsers.filter((user) => user.causeIds.includes(project.causeId) && user.fundingWeight > 0); const actor = random.weighted(supporters, (user) => user.fundingWeight); + const alignment = alignments.find((item) => item.projectId === project.id); + if (!alignment) throw new Error(`impossible fund-project: ${project.id} has no alignment action`); + add({ type: 'fund-project', actorUserId: actor.id, causeId: project.causeId, projectId: project.id, dependsOn: [createProjects.get(project.id)!.id, alignment.id] }); + } + const deposits: PlannedAction[] = []; + for (let index = 0; index < countByType['deposit-note']; index++) { const actor = random.weighted(activeUsers, (user) => user.activityWeight); deposits.push(add({ type: 'deposit-note', actorUserId: actor.id, noteId: `note-${String(index + 1).padStart(4, '0')}`, dependsOn: [] })); } + const delegations: PlannedAction[] = []; + for (let index = 0; index < countByType['delegate-note']; index++) { const deposit = deposits[index % deposits.length]; const delegate = random.pick(delegates.filter((user) => user.id !== deposit.actorUserId)); delegations.push(add({ type: 'delegate-note', actorUserId: deposit.actorUserId, noteId: deposit.noteId, delegateUserId: delegate.id, dependsOn: [deposit.id] })); } + for (let index = 0; index < countByType['revoke-delegation']; index++) { const delegation = delegations[index % delegations.length]; add({ type: 'revoke-delegation', actorUserId: delegation.actorUserId, noteId: delegation.noteId, dependsOn: [delegation.id] }); } + + validatePlannedActions(manifest, statements, users, projects, actions); + const writesByType = Object.fromEntries(manifest.actionRules.map((rule) => [rule.type, actions.filter((action) => action.type === rule.type).length])) as Record; + const estimatedGasByType = Object.fromEntries(Object.entries(writesByType).map(([type, count]) => [type, count * GAS_UNITS[type as CampaignActionType]])) as Record; + return { version: CAMPAIGN_PLAN_VERSION, campaignId: manifest.campaign.id, deterministicSeed: manifest.campaign.deterministicSeed, manifestFingerprint: sha256(stableJson(manifest)), statements, users, projects, actions, estimate: { writesByType, totalWrites: actions.length, estimatedGasByType, estimatedTotalGas: Object.values(estimatedGasByType).reduce((sum, value) => sum + value, 0), assumptions: { gasUnitsPerWrite: GAS_UNITS, paymentTokenUnitsPerFunding: 100 }, estimatedPaymentTokenUnits: writesByType['fund-project'] * 100 } }; +} + +export function validatePlannedActions(manifest: CampaignManifestV1, statements: PlannedStatement[], users: PlannedUser[], projects: PlannedProject[], actions: PlannedAction[]): void { + const actionById = new Map(actions.map((action) => [action.id, action])); + const statementIds = new Set(statements.map((item) => item.id)); const userIds = new Set(users.map((item) => item.id)); const projectIds = new Set(projects.map((item) => item.id)); + if (actionById.size !== actions.length) throw new Error('planned action IDs must be unique'); + for (const action of actions) { + if (action.actorUserId !== null && !userIds.has(action.actorUserId)) throw new Error(`${action.id} has missing actor ${action.actorUserId}`); + if (action.statementId && !statementIds.has(action.statementId)) throw new Error(`${action.id} has missing statement ${action.statementId}`); + if (action.projectId && !projectIds.has(action.projectId)) throw new Error(`${action.id} has missing project ${action.projectId}`); + for (const dependencyId of action.dependsOn) { const dependency = actionById.get(dependencyId); if (!dependency || dependency.sequence >= action.sequence) throw new Error(`${action.id} has impossible dependency ${dependencyId}`); } + const rule = manifest.actionRules.find((item) => item.type === action.type)!; + for (const prerequisite of rule.prerequisites) if (!action.dependsOn.some((id) => actionById.get(id)?.type === prerequisite)) throw new Error(`${action.id} is missing ${prerequisite} prerequisite`); + } + for (const rule of manifest.actionRules) { const count = actions.filter((action) => action.type === rule.type).length; if (count < rule.targetCount.min || count > rule.targetCount.max) throw new Error(`${rule.type} planned count ${count} is outside target ${rule.targetCount.min}-${rule.targetCount.max}`); } + if (users.some((user) => user.causeIds.length < 1 || user.causeIds.length > 3)) throw new Error('user cause assignment is outside 1-3 causes'); + if (new Set(users.map((user) => user.walletSlot)).size !== users.length) throw new Error('planned wallet slots must be unique'); +} + +async function writePlanArtifacts(manifest: CampaignManifestV1, plan: CampaignPlan, outputDirectory: string): Promise { + const writeJson = async (relativePath: string, value: JsonValue | object): Promise => { const target = path.join(outputDirectory, relativePath); await mkdir(path.dirname(target), { recursive: true }); await writeFile(target, `${JSON.stringify(value, null, 2)}\n`); }; + await writeJson(manifest.artifactLayout.manifest, manifest); + await writeJson(manifest.artifactLayout.statementCatalog, { version: plan.version, campaignId: plan.campaignId, statements: plan.statements }); + await writeJson(manifest.artifactLayout.assignments, { version: plan.version, campaignId: plan.campaignId, users: plan.users, projects: plan.projects }); + await writeJson(manifest.artifactLayout.walletAddresses, { version: plan.version, campaignId: plan.campaignId, wallets: plan.users.map(({ id, walletSlot }) => ({ userId: id, walletSlot, address: null, status: 'unprovisioned' })) }); + await writeJson(manifest.artifactLayout.actionPlan, { version: plan.version, campaignId: plan.campaignId, manifestFingerprint: plan.manifestFingerprint, actions: plan.actions, estimate: plan.estimate }); +} + +async function main(): Promise { + const directory = path.dirname(fileURLToPath(import.meta.url)); + const manifestPath = process.argv[2] ?? path.join(directory, 'campaigns/medium-realistic-v1.json'); + const outputDirectory = process.argv[3] ?? path.join(directory, 'output/campaigns/medium-realistic-v1'); + const manifest = JSON.parse(await readFile(manifestPath, 'utf8')) as CampaignManifestV1; + const plan = await buildCampaignPlan(manifest); await writePlanArtifacts(manifest, plan, outputDirectory); + console.log(`Planned ${plan.users.length} users, ${plan.statements.length} statements, ${plan.projects.length} projects, and ${plan.estimate.totalWrites} writes.`); + console.log(`Estimated gas: ${plan.estimate.estimatedTotalGas}; payment-token units: ${plan.estimate.estimatedPaymentTokenUnits}.`); + console.log(`Wrote planning artifacts to ${outputDirectory}`); +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) main().catch((error) => { console.error(error instanceof Error ? error.message : error); process.exitCode = 1; }); diff --git a/fake-data-generation/package.json b/fake-data-generation/package.json index 730b37b6..c256586a 100644 --- a/fake-data-generation/package.json +++ b/fake-data-generation/package.json @@ -17,6 +17,7 @@ "test:seed:worker-outputs": "tsx generateSeedWorkerOutputs.ts --verify", "test:seed:implication-regression": "tsx verifySeedImplicationEvaluations.ts", "gen:proliferation": "tsx generateProliferation.ts", + "gen:campaign:plan": "tsx campaignPlanner.ts", "gen:attesters": "tsx generateAttesters.ts", "gen:attestations": "tsx generateAttestations.ts", "gen:simulate": "tsx runSimulation.ts", diff --git a/fake-data-generation/test/campaignPlanner.test.ts b/fake-data-generation/test/campaignPlanner.test.ts new file mode 100644 index 00000000..d4551492 --- /dev/null +++ b/fake-data-generation/test/campaignPlanner.test.ts @@ -0,0 +1,43 @@ +import assert from 'node:assert/strict'; +import { readFile } from 'node:fs/promises'; +import test from 'node:test'; +import type { CampaignManifestV1 } from '../campaignSchema.js'; +import { buildCampaignPlan, validatePlannedActions } from '../campaignPlanner.js'; + +async function loadManifest(): Promise { + return JSON.parse(await readFile(new URL('../campaigns/medium-realistic-v1.json', import.meta.url), 'utf8')) as CampaignManifestV1; +} + +test('planner emits an identical, complete plan for the same seed', async () => { + const manifest = await loadManifest(); + const first = await buildCampaignPlan(manifest); const second = await buildCampaignPlan(manifest); + assert.deepEqual(first, second); + assert.equal(first.users.length, 100); assert.equal(first.statements.length, 46); + assert.ok(first.projects.length >= 18 && first.projects.length <= 26); + assert.ok(first.estimate.totalWrites >= 1_000 && first.estimate.totalWrites <= 3_000); + assert.equal(first.actions.length, first.estimate.totalWrites); + assert.equal(first.users.some((user) => 'privateKey' in user || 'address' in user), false); +}); + +test('planner assigns overlapping, uneven causes and respects persona bounds', async () => { + const manifest = await loadManifest(); const plan = await buildCampaignPlan(manifest); + const memberships = Object.fromEntries(manifest.causes.map((cause) => [cause.id, plan.users.filter((user) => user.causeIds.includes(cause.id)).length])); + assert.ok(new Set(Object.values(memberships)).size >= 5, JSON.stringify(memberships)); + assert.ok(plan.users.some((user) => user.causeIds.length > 1)); + for (const user of plan.users) { const persona = manifest.personas.find((item) => item.id === user.personaId)!; assert.ok(user.causeIds.length >= persona.causesPerUser.min && user.causeIds.length <= persona.causesPerUser.max); } +}); + +test('all implication actions use accepted bridge-role pairs', async () => { + const plan = await buildCampaignPlan(await loadManifest()); const statements = new Map(plan.statements.map((statement) => [statement.id, statement])); + for (const action of plan.actions.filter((item) => item.type === 'attest-implication')) { + assert.equal(action.implication?.evidence, 'accepted-bridge-role-pair'); + assert.match(statements.get(action.implication!.fromStatementId)!.role!, /^modified-(left|right)$/); + assert.equal(statements.get(action.implication!.toStatementId)!.role, 'commonality'); + } +}); + +test('validation rejects an action whose prerequisite points forward', async () => { + const manifest = await loadManifest(); const plan = await buildCampaignPlan(manifest); const invalid = structuredClone(plan.actions); + invalid[0].dependsOn = [invalid[1].id]; + assert.throws(() => validatePlannedActions(manifest, plan.statements, plan.users, plan.projects, invalid), /impossible dependency/); +}); From 94aa986deed6731a3e85762928522f3cb35031a2 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 13:49:28 -0400 Subject: [PATCH 04/19] Separate campaign chain environments --- fake-data-generation/README.md | 2 + .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignEnvironment.ts | 149 ++++++++++++++++++ .../test/campaignEnvironment.test.ts | 63 ++++++++ 4 files changed, 215 insertions(+), 1 deletion(-) create mode 100644 fake-data-generation/campaignEnvironment.ts create mode 100644 fake-data-generation/test/campaignEnvironment.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 976509f4..749772a8 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -13,6 +13,8 @@ The rest of this README is the **generative simulation** (jobs 1, 2, 4): randoml The medium realistic campaign has a separate planning-only command: `npm run gen:campaign:plan`. It deterministically writes the selected statement catalog, unprovisioned wallet slots, persona/cause/project assignments, complete prerequisite-linked action graph, and gas/payment-token estimate under `output/campaigns/medium-realistic-v1/`. A later provisioning step binds public addresses to those slots; planning neither creates private keys nor connects to a chain. +Campaign execution uses the explicit environment boundary in `campaignEnvironment.ts`. Local mode is pinned to Hardhat chain 31337 and may use the legacy deploy/Hardhat-wallet/free-mint conveniences. Remote mode requires a committed deployment env manifest, an explicit non-Hardhat chain ID, existing bytecode at every required address, generated campaign wallets, and transfer-only payment-token provisioning. Deployment and provisioning adapters are validated against those policies before they can run; remote execution must also carry a separate mutation-confirmation flag when it is added. The existing `gen:tiny`/`gen:small`/`gen:medium`/`gen:large` commands remain local-only legacy simulation paths and are not remote campaign commands. + ## Overview The generative testing suite consists of: diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index af671891..2a671585 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -179,7 +179,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 1. **[x] Freeze the v1 campaign schema and world.** [`campaignSchema.ts`](./campaignSchema.ts) defines and validates the versioned contract; [`campaigns/medium-realistic-v1.json`](./campaigns/medium-realistic-v1.json) selects 10 causes and 46 accepted statements and fixes the 100-user persona mix, uneven activity, action prerequisites/targets, artifact layout, synthetic-data label, and deterministic PRNG seed. Tests resolve every statement reference against accepted seed content and enforce the target shape. This remains planning-only and contains no wallets or transactions. 2. **[x] Build a deterministic local campaign planner.** [`campaignPlanner.ts`](./campaignPlanner.ts) expands the manifest into stable statement fingerprints, 100 persona/cause assignments, unprovisioned wallet slots, projects, and a prerequisite-linked action graph before any chain connection. `npm run gen:campaign:plan` writes the planning artifacts plus per-action write/gas and payment-token estimates. Tests cover deterministic replay, distribution bounds, accepted bridge-role implications, and impossible dependencies; validation also rejects missing/excluded statements and out-of-range action counts. -3. **[ ] Split local assumptions from reusable execution.** Introduce explicit chain configuration and adapters for deployment/provisioning. Keep existing tiny/demo/local commands working. Remote mode must refuse Hardhat keys, wrong chain IDs, missing bytecode, and implicit deployment/minting. +3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[ ] Add resumable, budgeted execution.** Persist action state and transaction hashes; add bounded concurrency, pacing, retry classification, stop/resume, transaction caps, and native-token budget enforcement. Prove interruption and resume locally. 5. **[ ] Make simulated behavior persona- and cause-aware.** Replace random implication truth and mechanically unrelated selections in the campaign path. Generate plausible project, belief, funding, alignment, and delegation histories from the manifest. 6. **[ ] Build transaction-to-indexer deep reconciliation.** Cover every selected action type, derived totals, duplicates, and settling-window behavior. Produce machine-readable results plus a concise human report. diff --git a/fake-data-generation/campaignEnvironment.ts b/fake-data-generation/campaignEnvironment.ts new file mode 100644 index 00000000..47bfb6fd --- /dev/null +++ b/fake-data-generation/campaignEnvironment.ts @@ -0,0 +1,149 @@ +import { readFile } from 'node:fs/promises'; +import { createPublicClient, http, isAddress, type Address, type Hex } from 'viem'; +import { privateKeyToAccount } from 'viem/accounts'; + +export const LOCAL_HARDHAT_CHAIN_ID = 31_337; +export const BASE_SEPOLIA_CHAIN_ID = 84_532; + +export const CAMPAIGN_CONTRACT_ENV_KEYS = { + beliefs: 'BELIEFS_CONTRACT_ADDRESS', + implications: 'IMPLICATIONS_CONTRACT_ADDRESS', + alignmentAttestations: 'ALIGNMENT_ATTESTATIONS_CONTRACT_ADDRESS', + delegatableNotes: 'DELEGATABLE_NOTES_CONTRACT_ADDRESS', + projectFactory: 'PROJECT_FACTORY_ADDRESS', + paymentToken: 'PAYMENT_TOKEN_ADDRESS', + publishedData: 'PUBLISHED_DATA_CONTRACT_ADDRESS', + mutableRefUpdater: 'MUTABLE_REF_UPDATER_CONTRACT_ADDRESS', +} as const; + +export type CampaignContractName = keyof typeof CAMPAIGN_CONTRACT_ENV_KEYS; +export type CampaignContracts = Record; + +interface CampaignEnvironmentBase { + rpcUrl: string; + expectedChainId: number; + contracts: CampaignContracts; +} + +export interface LocalCampaignEnvironment extends CampaignEnvironmentBase { + mode: 'local'; + deployment: { strategy: 'existing-or-deploy' }; + provisioning: { walletSource: 'generated-or-hardhat'; paymentTokenStrategy: 'transfer-or-mint' }; +} + +export interface RemoteCampaignEnvironment extends CampaignEnvironmentBase { + mode: 'remote'; + deployment: { strategy: 'existing-only' }; + provisioning: { walletSource: 'generated-only'; paymentTokenStrategy: 'transfer-only' }; + mutationConfirmed: boolean; +} + +export type CampaignEnvironment = LocalCampaignEnvironment | RemoteCampaignEnvironment; + +export interface CampaignChainAdapter { + getChainId(): Promise; + getBytecode(address: Address): Promise; +} + +export interface CampaignWalletBinding { + walletSlot: string; + address: Address; + privateKey: Hex; + source: 'generated' | 'hardhat'; +} + +export interface CampaignDeploymentAdapter { + readonly strategy: CampaignEnvironment['deployment']['strategy']; + prepare(environment: CampaignEnvironment): Promise; +} + +export interface CampaignProvisioningAdapter { + readonly walletSource: CampaignEnvironment['provisioning']['walletSource']; + readonly paymentTokenStrategy: CampaignEnvironment['provisioning']['paymentTokenStrategy']; + provision(environment: CampaignEnvironment, wallets: readonly CampaignWalletBinding[]): Promise; +} + +export interface CampaignPreflightResult { + mode: CampaignEnvironment['mode']; + chainId: number; + checkedContracts: CampaignContractName[]; +} + +export function createCampaignChainAdapter(rpcUrl: string): CampaignChainAdapter { + const client = createPublicClient({ transport: http(rpcUrl) }); + return { + getChainId: () => client.getChainId(), + getBytecode: (address) => client.getBytecode({ address }), + }; +} + +function parseEnvFile(contents: string): Record { + const values: Record = {}; + for (const rawLine of contents.split(/\r?\n/u)) { + const line = rawLine.trim(); + if (!line || line.startsWith('#')) continue; + const separator = line.indexOf('='); + if (separator < 1) continue; + const key = line.slice(0, separator).trim(); + const rawValue = line.slice(separator + 1).trim(); + values[key] = rawValue.replace(/^(?:"([\s\S]*)"|'([\s\S]*)')$/u, '$1$2'); + } + return values; +} + +function requireContracts(values: Record): CampaignContracts { + return Object.fromEntries(Object.entries(CAMPAIGN_CONTRACT_ENV_KEYS).map(([name, envKey]) => { + const value = values[envKey]; + if (!value) throw new Error(`${envKey} is missing from the deployment manifest`); + if (!isAddress(value)) throw new Error(`${envKey} is not a valid address`); + return [name, value]; + })) as unknown as CampaignContracts; +} + +export async function loadCampaignEnvironment(input: { + mode: 'local' | 'remote'; + rpcUrl: string; + expectedChainId: number; + deploymentEnvPath: string; + mutationConfirmed?: boolean; +}): Promise { + if (!input.rpcUrl.trim()) throw new Error('campaign RPC URL is required'); + if (!Number.isSafeInteger(input.expectedChainId) || input.expectedChainId <= 0) throw new Error('expected chain ID must be a positive integer'); + const contracts = requireContracts(parseEnvFile(await readFile(input.deploymentEnvPath, 'utf8'))); + if (input.mode === 'local') { + if (input.expectedChainId !== LOCAL_HARDHAT_CHAIN_ID) throw new Error(`local campaign mode requires Hardhat chain ${LOCAL_HARDHAT_CHAIN_ID}`); + return { mode: 'local', rpcUrl: input.rpcUrl, expectedChainId: input.expectedChainId, contracts, deployment: { strategy: 'existing-or-deploy' }, provisioning: { walletSource: 'generated-or-hardhat', paymentTokenStrategy: 'transfer-or-mint' } }; + } + if (input.expectedChainId === LOCAL_HARDHAT_CHAIN_ID) throw new Error('remote campaign mode refuses the local Hardhat chain ID'); + return { mode: 'remote', rpcUrl: input.rpcUrl, expectedChainId: input.expectedChainId, contracts, mutationConfirmed: input.mutationConfirmed === true, deployment: { strategy: 'existing-only' }, provisioning: { walletSource: 'generated-only', paymentTokenStrategy: 'transfer-only' } }; +} + +export function validateCampaignAdapters(environment: CampaignEnvironment, deployment: CampaignDeploymentAdapter, provisioning: CampaignProvisioningAdapter): void { + if (deployment.strategy !== environment.deployment.strategy) throw new Error(`deployment adapter strategy ${deployment.strategy} violates ${environment.mode} campaign policy ${environment.deployment.strategy}`); + if (provisioning.walletSource !== environment.provisioning.walletSource) throw new Error(`wallet source ${provisioning.walletSource} violates ${environment.mode} campaign policy ${environment.provisioning.walletSource}`); + if (provisioning.paymentTokenStrategy !== environment.provisioning.paymentTokenStrategy) throw new Error(`payment-token strategy ${provisioning.paymentTokenStrategy} violates ${environment.mode} campaign policy ${environment.provisioning.paymentTokenStrategy}`); +} + +export function validateCampaignWallets(environment: CampaignEnvironment, wallets: readonly CampaignWalletBinding[], hardhatPrivateKeys: readonly Hex[]): void { + const hardhatAddresses = new Set(hardhatPrivateKeys.map((key) => privateKeyToAccount(key).address.toLowerCase())); + const slots = new Set(); + const addresses = new Set(); + for (const wallet of wallets) { + if (slots.has(wallet.walletSlot)) throw new Error(`duplicate wallet slot ${wallet.walletSlot}`); + if (addresses.has(wallet.address.toLowerCase())) throw new Error(`duplicate wallet address ${wallet.address}`); + slots.add(wallet.walletSlot); addresses.add(wallet.address.toLowerCase()); + if (privateKeyToAccount(wallet.privateKey).address.toLowerCase() !== wallet.address.toLowerCase()) throw new Error(`private key does not match address for ${wallet.walletSlot}`); + if (environment.mode === 'remote' && (wallet.source === 'hardhat' || hardhatAddresses.has(wallet.address.toLowerCase()))) throw new Error(`remote campaign refuses Hardhat wallet ${wallet.walletSlot}`); + } +} + +export async function preflightCampaignEnvironment(environment: CampaignEnvironment, chain: CampaignChainAdapter): Promise { + const chainId = await chain.getChainId(); + if (chainId !== environment.expectedChainId) throw new Error(`wrong chain ID: expected ${environment.expectedChainId}, received ${chainId}`); + const checkedContracts = Object.keys(environment.contracts) as CampaignContractName[]; + for (const name of checkedContracts) { + const code = await chain.getBytecode(environment.contracts[name]); + if (!code || code === '0x') throw new Error(`missing contract bytecode for ${name} at ${environment.contracts[name]}`); + } + return { mode: environment.mode, chainId, checkedContracts }; +} diff --git a/fake-data-generation/test/campaignEnvironment.test.ts b/fake-data-generation/test/campaignEnvironment.test.ts new file mode 100644 index 00000000..84c7a99d --- /dev/null +++ b/fake-data-generation/test/campaignEnvironment.test.ts @@ -0,0 +1,63 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { privateKeyToAccount } from 'viem/accounts'; +import { + BASE_SEPOLIA_CHAIN_ID, CAMPAIGN_CONTRACT_ENV_KEYS, LOCAL_HARDHAT_CHAIN_ID, + loadCampaignEnvironment, preflightCampaignEnvironment, validateCampaignAdapters, validateCampaignWallets, + type CampaignDeploymentAdapter, type CampaignProvisioningAdapter, +} from '../campaignEnvironment.js'; + +const hardhatKey = '0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80' as const; +const generatedKey = '0x59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d' as const; + +async function withManifest(run: (manifestPath: string) => Promise): Promise { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-environment-')); + const manifestPath = path.join(directory, 'deployment.env'); + const lines = Object.values(CAMPAIGN_CONTRACT_ENV_KEYS).map((key, index) => `${key}=0x${String(index + 1).padStart(40, '0')}`); + await writeFile(manifestPath, `${lines.join('\n')}\n`); + try { await run(manifestPath); } finally { await rm(directory, { recursive: true, force: true }); } +} + +test('local environment makes legacy-only assumptions explicit', async () => withManifest(async (deploymentEnvPath) => { + const environment = await loadCampaignEnvironment({ mode: 'local', rpcUrl: 'http://127.0.0.1:8545', expectedChainId: LOCAL_HARDHAT_CHAIN_ID, deploymentEnvPath }); + assert.deepEqual(environment.deployment, { strategy: 'existing-or-deploy' }); + assert.deepEqual(environment.provisioning, { walletSource: 'generated-or-hardhat', paymentTokenStrategy: 'transfer-or-mint' }); +})); + +test('remote environment is existing-deployment, generated-wallet, transfer-only', async () => withManifest(async (deploymentEnvPath) => { + const environment = await loadCampaignEnvironment({ mode: 'remote', rpcUrl: 'https://sepolia.base.org', expectedChainId: BASE_SEPOLIA_CHAIN_ID, deploymentEnvPath }); + assert.equal(environment.mutationConfirmed, false); + assert.deepEqual(environment.deployment, { strategy: 'existing-only' }); + assert.deepEqual(environment.provisioning, { walletSource: 'generated-only', paymentTokenStrategy: 'transfer-only' }); +})); + +test('remote mode refuses local chain IDs, implicit deployment, minting, and Hardhat wallets', async () => withManifest(async (deploymentEnvPath) => { + await assert.rejects(loadCampaignEnvironment({ mode: 'remote', rpcUrl: 'http://127.0.0.1:8545', expectedChainId: LOCAL_HARDHAT_CHAIN_ID, deploymentEnvPath }), /refuses the local Hardhat chain ID/); + const environment = await loadCampaignEnvironment({ mode: 'remote', rpcUrl: 'https://sepolia.base.org', expectedChainId: BASE_SEPOLIA_CHAIN_ID, deploymentEnvPath }); + const deployment = { strategy: 'existing-or-deploy', prepare: async () => environment.contracts } as CampaignDeploymentAdapter; + const safeProvisioning = { walletSource: 'generated-only', paymentTokenStrategy: 'transfer-only', provision: async () => undefined } as CampaignProvisioningAdapter; + assert.throws(() => validateCampaignAdapters(environment, deployment, safeProvisioning), /deployment adapter strategy/); + const safeDeployment = { strategy: 'existing-only', prepare: async () => environment.contracts } as CampaignDeploymentAdapter; + const mintingProvisioning = { walletSource: 'generated-only', paymentTokenStrategy: 'transfer-or-mint', provision: async () => undefined } as CampaignProvisioningAdapter; + assert.throws(() => validateCampaignAdapters(environment, safeDeployment, mintingProvisioning), /payment-token strategy/); + const address = privateKeyToAccount(hardhatKey).address; + assert.throws(() => validateCampaignWallets(environment, [{ walletSlot: 'wallet-user-001', address, privateKey: hardhatKey, source: 'hardhat' }], [hardhatKey]), /refuses Hardhat wallet/); +})); + +test('preflight rejects a wrong chain and missing bytecode', async () => withManifest(async (deploymentEnvPath) => { + const environment = await loadCampaignEnvironment({ mode: 'remote', rpcUrl: 'https://sepolia.base.org', expectedChainId: BASE_SEPOLIA_CHAIN_ID, deploymentEnvPath }); + await assert.rejects(preflightCampaignEnvironment(environment, { getChainId: async () => 1, getBytecode: async () => '0x01' }), /wrong chain ID/); + await assert.rejects(preflightCampaignEnvironment(environment, { getChainId: async () => BASE_SEPOLIA_CHAIN_ID, getBytecode: async () => '0x' }), /missing contract bytecode/); + const result = await preflightCampaignEnvironment(environment, { getChainId: async () => BASE_SEPOLIA_CHAIN_ID, getBytecode: async () => '0x01' }); + assert.equal(result.checkedContracts.length, Object.keys(CAMPAIGN_CONTRACT_ENV_KEYS).length); +})); + +test('wallet validation checks key ownership and permits generated remote wallets', async () => withManifest(async (deploymentEnvPath) => { + const environment = await loadCampaignEnvironment({ mode: 'remote', rpcUrl: 'https://sepolia.base.org', expectedChainId: BASE_SEPOLIA_CHAIN_ID, deploymentEnvPath }); + const address = privateKeyToAccount(generatedKey).address; + validateCampaignWallets(environment, [{ walletSlot: 'wallet-user-001', address, privateKey: generatedKey, source: 'generated' }], [hardhatKey]); + assert.throws(() => validateCampaignWallets(environment, [{ walletSlot: 'wallet-user-001', address, privateKey: hardhatKey, source: 'generated' }], [hardhatKey]), /does not match address/); +})); From 6a12e4bf3ef2078db80bd18a28d980c4e891f4a5 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 13:56:59 -0400 Subject: [PATCH 05/19] Add resumable campaign execution engine --- fake-data-generation/README.md | 2 + .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignExecutor.ts | 199 ++++++++++++++++++ .../test/campaignExecutor.test.ts | 82 ++++++++ 4 files changed, 284 insertions(+), 1 deletion(-) create mode 100644 fake-data-generation/campaignExecutor.ts create mode 100644 fake-data-generation/test/campaignExecutor.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 749772a8..ab84631a 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -15,6 +15,8 @@ The medium realistic campaign has a separate planning-only command: `npm run gen Campaign execution uses the explicit environment boundary in `campaignEnvironment.ts`. Local mode is pinned to Hardhat chain 31337 and may use the legacy deploy/Hardhat-wallet/free-mint conveniences. Remote mode requires a committed deployment env manifest, an explicit non-Hardhat chain ID, existing bytecode at every required address, generated campaign wallets, and transfer-only payment-token provisioning. Deployment and provisioning adapters are validated against those policies before they can run; remote execution must also carry a separate mutation-confirmation flag when it is added. The existing `gen:tiny`/`gen:small`/`gen:medium`/`gen:large` commands remain local-only legacy simulation paths and are not remote campaign commands. +The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. No CLI or real contract adapter is exposed yet: this layer is deliberately tested against a local fake chain before the campaign behavior and contract bindings are added. + ## Overview The generative testing suite consists of: diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 2a671585..602ec23a 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -180,7 +180,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 1. **[x] Freeze the v1 campaign schema and world.** [`campaignSchema.ts`](./campaignSchema.ts) defines and validates the versioned contract; [`campaigns/medium-realistic-v1.json`](./campaigns/medium-realistic-v1.json) selects 10 causes and 46 accepted statements and fixes the 100-user persona mix, uneven activity, action prerequisites/targets, artifact layout, synthetic-data label, and deterministic PRNG seed. Tests resolve every statement reference against accepted seed content and enforce the target shape. This remains planning-only and contains no wallets or transactions. 2. **[x] Build a deterministic local campaign planner.** [`campaignPlanner.ts`](./campaignPlanner.ts) expands the manifest into stable statement fingerprints, 100 persona/cause assignments, unprovisioned wallet slots, projects, and a prerequisite-linked action graph before any chain connection. `npm run gen:campaign:plan` writes the planning artifacts plus per-action write/gas and payment-token estimates. Tests cover deterministic replay, distribution bounds, accepted bridge-role implications, and impossible dependencies; validation also rejects missing/excluded statements and out-of-range action counts. 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. -4. **[ ] Add resumable, budgeted execution.** Persist action state and transaction hashes; add bounded concurrency, pacing, retry classification, stop/resume, transaction caps, and native-token budget enforcement. Prove interruption and resume locally. +4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[ ] Make simulated behavior persona- and cause-aware.** Replace random implication truth and mechanically unrelated selections in the campaign path. Generate plausible project, belief, funding, alignment, and delegation histories from the manifest. 6. **[ ] Build transaction-to-indexer deep reconciliation.** Cover every selected action type, derived totals, duplicates, and settling-window behavior. Produce machine-readable results plus a concise human report. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. diff --git a/fake-data-generation/campaignExecutor.ts b/fake-data-generation/campaignExecutor.ts new file mode 100644 index 00000000..57256c29 --- /dev/null +++ b/fake-data-generation/campaignExecutor.ts @@ -0,0 +1,199 @@ +import { mkdir, readFile, rename, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import type { Hex } from 'viem'; +import type { PlannedAction } from './campaignPlanner.js'; + +export const CAMPAIGN_EXECUTION_VERSION = 'commonality-campaign-execution-v1' as const; + +export type CampaignActionStatus = 'planned' | 'submitted' | 'mined' | 'failed'; + +export interface CampaignReceipt { + status: 'success' | 'reverted'; + gasUsed: bigint; + effectiveGasPrice: bigint; +} + +export interface CampaignExecutionAdapter { + estimateNativeCost(action: PlannedAction): Promise; + submit(action: PlannedAction): Promise; + getReceipt(transactionHash: Hex): Promise; + classifyError(error: unknown): { retryable: boolean; category: string; message: string }; +} + +export interface CampaignActionExecution { + actionId: string; + status: CampaignActionStatus; + attempts: number; + transactionHash?: Hex; + gasUsed?: string; + nativeCost?: string; + failure?: { category: string; message: string }; +} + +export interface CampaignExecutionState { + version: typeof CAMPAIGN_EXECUTION_VERSION; + campaignId: string; + manifestFingerprint: string; + updatedAt: string; + actions: CampaignActionExecution[]; +} + +export interface CampaignExecutionOptions { + statePath: string; + concurrency: number; + pacingMs: number; + maxRetries: number; + retryBackoffMs: number; + transactionCap: number; + nativeTokenBudget: bigint; + shouldStop?: () => boolean; + now?: () => Date; + sleep?: (milliseconds: number) => Promise; +} + +export interface CampaignExecutionSummary { + stopped: boolean; + mined: number; + failed: number; + submitted: number; + planned: number; + transactions: number; + nativeCost: bigint; +} + +function defaultSleep(milliseconds: number): Promise { + return new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +async function persistState(statePath: string, state: CampaignExecutionState): Promise { + await mkdir(path.dirname(statePath), { recursive: true }); + const temporaryPath = `${statePath}.tmp`; + await writeFile(temporaryPath, `${JSON.stringify(state, null, 2)}\n`); + await rename(temporaryPath, statePath); +} + +async function loadState(statePath: string): Promise { + try { + return JSON.parse(await readFile(statePath, 'utf8')) as CampaignExecutionState; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return null; + throw error; + } +} + +function summarize(state: CampaignExecutionState, stopped: boolean): CampaignExecutionSummary { + const count = (status: CampaignActionStatus): number => state.actions.filter((action) => action.status === status).length; + return { + stopped, + mined: count('mined'), failed: count('failed'), submitted: count('submitted'), planned: count('planned'), + transactions: state.actions.filter((action) => action.transactionHash).length, + nativeCost: state.actions.reduce((sum, action) => sum + BigInt(action.nativeCost ?? 0), 0n), + }; +} + +export async function executeCampaignPlan(input: { + campaignId: string; + manifestFingerprint: string; + actions: readonly PlannedAction[]; + adapter: CampaignExecutionAdapter; + options: CampaignExecutionOptions; +}): Promise { + const { actions, adapter, options } = input; + if (!Number.isInteger(options.concurrency) || options.concurrency < 1) throw new Error('campaign concurrency must be a positive integer'); + if (!Number.isInteger(options.transactionCap) || options.transactionCap < 0) throw new Error('campaign transaction cap must be a non-negative integer'); + if (options.nativeTokenBudget < 0n) throw new Error('campaign native-token budget must be non-negative'); + const now = options.now ?? (() => new Date()); + const sleep = options.sleep ?? defaultSleep; + let state = await loadState(options.statePath); + if (!state) { + state = { version: CAMPAIGN_EXECUTION_VERSION, campaignId: input.campaignId, manifestFingerprint: input.manifestFingerprint, updatedAt: now().toISOString(), actions: actions.map((action) => ({ actionId: action.id, status: 'planned', attempts: 0 })) }; + await persistState(options.statePath, state); + } + if (state.version !== CAMPAIGN_EXECUTION_VERSION || state.campaignId !== input.campaignId || state.manifestFingerprint !== input.manifestFingerprint) throw new Error('execution state does not match this campaign plan'); + if (state.actions.length !== actions.length || state.actions.some((item, index) => item.actionId !== actions[index].id)) throw new Error('execution state action list does not match this campaign plan'); + + const records = new Map(state.actions.map((record) => [record.actionId, record])); + let reservedNativeCost = state.actions.reduce((sum, record) => sum + BigInt(record.nativeCost ?? 0), 0n); + let transactionCount = state.actions.filter((record) => record.transactionHash).length; + let lastSubmissionAt = 0; + let stopped = false; + let stateWrite = Promise.resolve(); + let submissionLock = Promise.resolve(); + const save = async (): Promise => { + state!.updatedAt = now().toISOString(); + stateWrite = stateWrite.then(() => persistState(options.statePath, state!)); + await stateWrite; + }; + + const submitWithinBudget = async (action: PlannedAction, record: CampaignActionExecution, estimate: bigint): Promise => { + const previous = submissionLock; + let release = (): void => undefined; + submissionLock = new Promise((resolve) => { release = resolve; }); + await previous; + try { + if (transactionCount >= options.transactionCap) throw new Error(`campaign transaction cap ${options.transactionCap} would be exceeded`); + if (reservedNativeCost + estimate > options.nativeTokenBudget) throw new Error(`campaign native-token budget ${options.nativeTokenBudget} would be exceeded`); + const pacingWait = Math.max(0, lastSubmissionAt + options.pacingMs - Date.now()); + if (pacingWait > 0) await sleep(pacingWait); + record.attempts += 1; + const transactionHash = await adapter.submit(action); + lastSubmissionAt = Date.now(); + record.status = 'submitted'; record.transactionHash = transactionHash; record.nativeCost = estimate.toString(); + transactionCount += 1; reservedNativeCost += estimate; + await save(); + } finally { release(); } + }; + + const run = async (action: PlannedAction): Promise => { + const record = records.get(action.id)!; + let receiptRetries = 0; + for (;;) { + if (options.shouldStop?.()) { stopped = true; return; } + if (record.status === 'submitted') { + try { + const receipt = await adapter.getReceipt(record.transactionHash!); + if (!receipt) return; + const previousCost = BigInt(record.nativeCost ?? 0); + const actualCost = receipt.gasUsed * receipt.effectiveGasPrice; + reservedNativeCost += actualCost - previousCost; + record.gasUsed = receipt.gasUsed.toString(); record.nativeCost = actualCost.toString(); + record.status = receipt.status === 'success' ? 'mined' : 'failed'; + if (receipt.status === 'reverted') record.failure = { category: 'contract-revert', message: 'transaction reverted' }; + await save(); + return; + } catch (error) { + const failure = adapter.classifyError(error); + receiptRetries += 1; + if (failure.retryable && receiptRetries <= options.maxRetries) { await sleep(options.retryBackoffMs * receiptRetries); continue; } + return; + } + } + try { + const estimate = await adapter.estimateNativeCost(action); + await submitWithinBudget(action, record, estimate); + } catch (error) { + if (error instanceof Error && error.message.startsWith('campaign ')) throw error; + const failure = adapter.classifyError(error); + if (failure.retryable && record.attempts <= options.maxRetries) { await sleep(options.retryBackoffMs * record.attempts); continue; } + record.status = 'failed'; record.failure = { category: failure.category, message: failure.message }; + await save(); + return; + } + } + }; + + while (!stopped) { + if (options.shouldStop?.()) { stopped = true; break; } + const ready = actions.filter((action) => { + const record = records.get(action.id)!; + if (record.status === 'mined' || record.status === 'failed') return false; + return record.status === 'submitted' || action.dependsOn.every((dependency) => records.get(dependency)?.status === 'mined'); + }); + if (ready.length === 0) break; + const batch = ready.slice(0, options.concurrency); + await Promise.all(batch.map(run)); + if (batch.every((action) => records.get(action.id)!.status === 'submitted')) break; + } + await stateWrite; + return summarize(state, stopped); +} diff --git a/fake-data-generation/test/campaignExecutor.test.ts b/fake-data-generation/test/campaignExecutor.test.ts new file mode 100644 index 00000000..91bce303 --- /dev/null +++ b/fake-data-generation/test/campaignExecutor.test.ts @@ -0,0 +1,82 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import type { Hex } from 'viem'; +import { executeCampaignPlan, type CampaignExecutionAdapter } from '../campaignExecutor.js'; +import type { PlannedAction } from '../campaignPlanner.js'; + +const actions: PlannedAction[] = [ + { id: 'action-1', sequence: 1, type: 'publish-statement', actorUserId: null, statementId: 'statement-1', dependsOn: [] }, + { id: 'action-2', sequence: 2, type: 'set-belief', actorUserId: 'user-1', statementId: 'statement-1', dependsOn: ['action-1'] }, + { id: 'action-3', sequence: 3, type: 'deposit-note', actorUserId: 'user-1', noteId: 'note-1', dependsOn: [] }, + { id: 'action-4', sequence: 4, type: 'delegate-note', actorUserId: 'user-1', noteId: 'note-1', delegateUserId: 'user-2', dependsOn: ['action-3'] }, +]; + +function adapter(submitted: string[], attempts = new Map()): CampaignExecutionAdapter { + return { + estimateNativeCost: async () => 10n, + submit: async (action) => { + attempts.set(action.id, (attempts.get(action.id) ?? 0) + 1); + if (action.id === 'action-3' && attempts.get(action.id) === 1) throw new Error('rate limited'); + submitted.push(action.id); + return `0x${action.sequence.toString(16).padStart(64, '0')}` as Hex; + }, + getReceipt: async () => ({ status: 'success', gasUsed: 2n, effectiveGasPrice: 3n }), + classifyError: (error) => ({ retryable: (error as Error).message === 'rate limited', category: 'rpc', message: (error as Error).message }), + }; +} + +test('persists interruption and resumes without resubmitting mined actions', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-execution-')); + const statePath = path.join(directory, 'execution.json'); + const submitted: string[] = []; + try { + const first = await executeCampaignPlan({ campaignId: 'test', manifestFingerprint: 'fingerprint', actions, adapter: adapter(submitted), options: { statePath, concurrency: 1, pacingMs: 0, maxRetries: 2, retryBackoffMs: 0, transactionCap: 10, nativeTokenBudget: 100n, shouldStop: () => submitted.length >= 2 } }); + assert.equal(first.stopped, true); assert.equal(first.mined, 1); assert.equal(first.submitted, 1); + const second = await executeCampaignPlan({ campaignId: 'test', manifestFingerprint: 'fingerprint', actions, adapter: adapter(submitted), options: { statePath, concurrency: 2, pacingMs: 0, maxRetries: 2, retryBackoffMs: 0, transactionCap: 10, nativeTokenBudget: 100n } }); + assert.equal(second.mined, 4); assert.equal(second.planned, 0); + assert.equal(new Set(submitted).size, 4); + const state = JSON.parse(await readFile(statePath, 'utf8')) as { actions: Array<{ actionId: string; status: string; attempts: number }> }; + assert.ok(state.actions.every((action) => action.status === 'mined')); + assert.equal(state.actions.find((action) => action.actionId === 'action-3')?.attempts, 2); + } finally { await rm(directory, { recursive: true, force: true }); } +}); + +test('fails closed before exceeding transaction or native-token budgets', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-budget-')); + const submitted: string[] = []; + try { + await assert.rejects(executeCampaignPlan({ campaignId: 'test', manifestFingerprint: 'fingerprint', actions, adapter: adapter(submitted), options: { statePath: path.join(directory, 'cap.json'), concurrency: 4, pacingMs: 0, maxRetries: 0, retryBackoffMs: 0, transactionCap: 1, nativeTokenBudget: 100n } }), /transaction cap/); + assert.equal(submitted.length, 1); + submitted.length = 0; + await assert.rejects(executeCampaignPlan({ campaignId: 'test', manifestFingerprint: 'fingerprint', actions, adapter: adapter(submitted), options: { statePath: path.join(directory, 'budget.json'), concurrency: 1, pacingMs: 0, maxRetries: 0, retryBackoffMs: 0, transactionCap: 10, nativeTokenBudget: 9n } }), /native-token budget/); + assert.equal(submitted.length, 0); + } finally { await rm(directory, { recursive: true, force: true }); } +}); + +test('refuses state from a different immutable plan', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-mismatch-')); + const statePath = path.join(directory, 'execution.json'); + try { + await executeCampaignPlan({ campaignId: 'test', manifestFingerprint: 'one', actions: actions.slice(0, 1), adapter: adapter([]), options: { statePath, concurrency: 1, pacingMs: 0, maxRetries: 0, retryBackoffMs: 0, transactionCap: 1, nativeTokenBudget: 10n } }); + await assert.rejects(executeCampaignPlan({ campaignId: 'test', manifestFingerprint: 'two', actions: actions.slice(0, 1), adapter: adapter([]), options: { statePath, concurrency: 1, pacingMs: 0, maxRetries: 0, retryBackoffMs: 0, transactionCap: 1, nativeTokenBudget: 10n } }), /does not match/); + } finally { await rm(directory, { recursive: true, force: true }); } +}); + +test('a receipt RPC error never causes duplicate transaction submission', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-receipt-')); + const submitted: string[] = []; + const fakeAdapter = adapter(submitted); + let receiptCalls = 0; + fakeAdapter.getReceipt = async () => { + receiptCalls += 1; + if (receiptCalls === 1) throw new Error('rate limited'); + return { status: 'success', gasUsed: 2n, effectiveGasPrice: 3n }; + }; + try { + const summary = await executeCampaignPlan({ campaignId: 'test', manifestFingerprint: 'fingerprint', actions: actions.slice(0, 1), adapter: fakeAdapter, options: { statePath: path.join(directory, 'execution.json'), concurrency: 1, pacingMs: 0, maxRetries: 1, retryBackoffMs: 0, transactionCap: 1, nativeTokenBudget: 10n } }); + assert.equal(summary.mined, 1); assert.deepEqual(submitted, ['action-1']); assert.equal(receiptCalls, 2); + } finally { await rm(directory, { recursive: true, force: true }); } +}); From 13dadc9af73b59f8d7a6f46328b056a1974ea50a Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 14:06:15 -0400 Subject: [PATCH 06/19] Make campaign behavior cause-aware --- fake-data-generation/README.md | 2 +- .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignPlanner.ts | 72 ++++++++++++++++--- .../test/campaignPlanner.test.ts | 45 ++++++++++++ 4 files changed, 108 insertions(+), 13 deletions(-) diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index ab84631a..379c165c 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -11,7 +11,7 @@ This directory is **four jobs**, not one “generate fake data” switch: The rest of this README is the **generative simulation** (jobs 1, 2, 4): randomly generated users and on-chain actions. Job 3 is the JSON + attester loop, not `runSimulation`. -The medium realistic campaign has a separate planning-only command: `npm run gen:campaign:plan`. It deterministically writes the selected statement catalog, unprovisioned wallet slots, persona/cause/project assignments, complete prerequisite-linked action graph, and gas/payment-token estimate under `output/campaigns/medium-realistic-v1/`. A later provisioning step binds public addresses to those slots; planning neither creates private keys nor connects to a chain. +The medium realistic campaign has a separate planning-only command: `npm run gen:campaign:plan`. It deterministically writes the selected statement catalog, unprovisioned wallet slots, persona/cause/project assignments, complete prerequisite-linked action graph, and gas/payment-token estimate under `output/campaigns/medium-realistic-v1/`. Its actions carry persona- and cause-aware intent: belief values and changes, project outcome text, outcome-backed alignments, skewed funding amounts (including deliberately unfunded projects), and shared-cause trust reasons for delegation/revocation histories. A later provisioning step binds public addresses to those slots; planning neither creates private keys nor connects to a chain. Campaign execution uses the explicit environment boundary in `campaignEnvironment.ts`. Local mode is pinned to Hardhat chain 31337 and may use the legacy deploy/Hardhat-wallet/free-mint conveniences. Remote mode requires a committed deployment env manifest, an explicit non-Hardhat chain ID, existing bytecode at every required address, generated campaign wallets, and transfer-only payment-token provisioning. Deployment and provisioning adapters are validated against those policies before they can run; remote execution must also carry a separate mutation-confirmation flag when it is added. The existing `gen:tiny`/`gen:small`/`gen:medium`/`gen:large` commands remain local-only legacy simulation paths and are not remote campaign commands. diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 602ec23a..872aca48 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -181,7 +181,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 2. **[x] Build a deterministic local campaign planner.** [`campaignPlanner.ts`](./campaignPlanner.ts) expands the manifest into stable statement fingerprints, 100 persona/cause assignments, unprovisioned wallet slots, projects, and a prerequisite-linked action graph before any chain connection. `npm run gen:campaign:plan` writes the planning artifacts plus per-action write/gas and payment-token estimates. Tests cover deterministic replay, distribution bounds, accepted bridge-role implications, and impossible dependencies; validation also rejects missing/excluded statements and out-of-range action counts. 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. -5. **[ ] Make simulated behavior persona- and cause-aware.** Replace random implication truth and mechanically unrelated selections in the campaign path. Generate plausible project, belief, funding, alignment, and delegation histories from the manifest. +5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. 6. **[ ] Build transaction-to-indexer deep reconciliation.** Cover every selected action type, derived totals, duplicates, and settling-window behavior. Produce machine-readable results plus a concise human report. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. diff --git a/fake-data-generation/campaignPlanner.ts b/fake-data-generation/campaignPlanner.ts index 2f37a9a4..8cf21732 100644 --- a/fake-data-generation/campaignPlanner.ts +++ b/fake-data-generation/campaignPlanner.ts @@ -30,6 +30,8 @@ export interface PlannedUser { export interface PlannedProject { id: string; + title: string; + outcome: string; causeId: string; founderUserId: string; statementIds: string[]; @@ -45,6 +47,10 @@ export interface PlannedAction { projectId?: string; noteId?: string; delegateUserId?: string; + belief?: 'believe' | 'disbelieve'; + amount?: number; + alignment?: 'supports-described-outcome'; + delegationBasis?: { sharedCauseIds: string[]; reason: 'shared-cause-trusted-role' }; implication?: { fromStatementId: string; toStatementId: string; evidence: 'accepted-bridge-role-pair' }; dependsOn: string[]; } @@ -63,7 +69,7 @@ export interface CampaignPlan { totalWrites: number; estimatedGasByType: Record; estimatedTotalGas: number; - assumptions: { gasUnitsPerWrite: Record; paymentTokenUnitsPerFunding: number }; + assumptions: { gasUnitsPerWrite: Record; paymentTokenBaseUnit: number }; estimatedPaymentTokenUnits: number; }; } @@ -140,6 +146,27 @@ function validateImplicationPair(from: PlannedStatement, to: PlannedStatement): if (!sameGroup || !approved) throw new Error(`unsuitable implication ${from.id} -> ${to.id}: pair lacks accepted bridge-role evidence`); } +function usersForCause(users: PlannedUser[], causeId: string): PlannedUser[] { + return users.filter((user) => user.causeIds.includes(causeId)); +} + +function chooseCauseAttester(users: PlannedUser[], causeId: string, random: Xoshiro128StarStar): PlannedUser { + const members = usersForCause(users, causeId); + return random.weighted(members, (user) => user.activityWeight * (user.roles.includes('attester') || user.roles.includes('mediator') ? 8 : 1)); +} + +function preferredStatements(user: PlannedUser, statements: PlannedStatement[]): PlannedStatement[] { + const inCauses = statements.filter((statement) => user.causeIds.includes(statement.causeId)); + if (!user.roles.includes('mediator')) return inCauses.filter((statement) => !statement.role?.startsWith('natural-')); + return inCauses.filter((statement) => statement.role === null || statement.role === 'commonality' || statement.role.startsWith('modified-')); +} + +function fundingAmount(user: PlannedUser, projectIndex: number, random: Xoshiro128StarStar): number { + const personaScale = Math.max(1, Math.round(user.fundingWeight * 100)); + const projectPopularity = Math.max(1, 6 - (projectIndex % 7)); + return personaScale * projectPopularity * random.integer(1, 3); +} + export async function buildCampaignPlan(manifest: CampaignManifestV1): Promise { validateCampaignManifest(manifest); const random = new Xoshiro128StarStar(manifest.campaign.deterministicSeed); @@ -172,7 +199,9 @@ export async function buildCampaignPlan(manifest: CampaignManifestV1): Promise statement.causeId === causeId); - return { id: `project-${String(index + 1).padStart(3, '0')}`, causeId, founderUserId: founder.id, statementIds: random.shuffle(candidates).slice(0, random.integer(1, Math.min(3, candidates.length))).map((statement) => statement.id) }; + const selected = random.shuffle(candidates).slice(0, random.integer(1, Math.min(3, candidates.length))); + const cause = manifest.causes.find((item) => item.id === causeId)!; + return { id: `project-${String(index + 1).padStart(3, '0')}`, title: `${cause.title}: ${selected[0].text}`, outcome: selected.map((statement) => statement.text).join(' '), causeId, founderUserId: founder.id, statementIds: selected.map((statement) => statement.id) }; }); const actions: PlannedAction[] = []; @@ -180,10 +209,14 @@ export async function buildCampaignPlan(manifest: CampaignManifestV1): Promise [statement.id, add({ type: 'publish-statement', actorUserId: null, causeId: statement.causeId, statementId: statement.id, dependsOn: [] })])); const causes = new Map(manifest.causes.map((cause) => [cause.id, add({ type: 'create-cause', actorUserId: random.pick(activeUsers.filter((user) => user.roles.includes('cause-founder') || user.roles.includes('power-user'))).id, causeId: cause.id, dependsOn: cause.statementRefs.map((ref) => publishes.get(statements.find((statement) => statementKey(statement.source) === statementKey(ref))!.id)!.id) })])); + const latestBelief = new Map(); for (let index = 0; index < countByType['set-belief']; index++) { const actor = random.weighted(activeUsers, (user) => user.activityWeight); - const statement = random.pick(statements.filter((item) => actor.causeIds.includes(item.causeId))); - add({ type: 'set-belief', actorUserId: actor.id, causeId: statement.causeId, statementId: statement.id, dependsOn: [publishes.get(statement.id)!.id] }); + const previous = index > Math.floor(countByType['set-belief'] * 0.92) && latestBelief.size > 0 ? random.pick([...latestBelief.values()]) : undefined; + const statement = previous ? statements.find((item) => item.id === previous.statementId)! : random.pick(preferredStatements(actor, statements)); + const belief = previous ? (previous.belief === 'believe' ? 'disbelieve' : 'believe') : (random.next() < 0.94 ? 'believe' : 'disbelieve'); + const action = add({ type: 'set-belief', actorUserId: previous?.actorUserId ?? actor.id, causeId: statement.causeId, statementId: statement.id, belief, dependsOn: previous ? [publishes.get(statement.id)!.id, previous.id] : [publishes.get(statement.id)!.id] }); + latestBelief.set(`${action.actorUserId}/${statement.id}`, action); } const implicationPairs = manifest.causes.flatMap((cause) => { const groupStatements = statements.filter((statement) => statement.causeId === cause.id); @@ -193,30 +226,41 @@ export async function buildCampaignPlan(manifest: CampaignManifestV1): Promise 0 && implicationPairs.length === 0) throw new Error('no suitable accepted implication pairs are available'); for (let index = 0; index < countByType['attest-implication']; index++) { const pair = implicationPairs[index % implicationPairs.length]; validateImplicationPair(pair.from, pair.to); - add({ type: 'attest-implication', actorUserId: random.pick(attesters).id, causeId: pair.from.causeId, implication: { fromStatementId: pair.from.id, toStatementId: pair.to.id, evidence: 'accepted-bridge-role-pair' }, dependsOn: [publishes.get(pair.from.id)!.id, publishes.get(pair.to.id)!.id] }); + add({ type: 'attest-implication', actorUserId: chooseCauseAttester(activeUsers, pair.from.causeId, random).id, causeId: pair.from.causeId, implication: { fromStatementId: pair.from.id, toStatementId: pair.to.id, evidence: 'accepted-bridge-role-pair' }, dependsOn: [publishes.get(pair.from.id)!.id, publishes.get(pair.to.id)!.id] }); } const createProjects = new Map(projects.map((project) => [project.id, add({ type: 'create-project', actorUserId: project.founderUserId, causeId: project.causeId, projectId: project.id, dependsOn: [causes.get(project.causeId)!.id] })])); const alignments: PlannedAction[] = []; for (let index = 0; index < countByType['attest-alignment']; index++) { const project = projects[index % projects.length]; const statementId = project.statementIds[index % project.statementIds.length]; - alignments.push(add({ type: 'attest-alignment', actorUserId: random.pick(attesters).id, causeId: project.causeId, projectId: project.id, statementId, dependsOn: [createProjects.get(project.id)!.id, publishes.get(statementId)!.id] })); + alignments.push(add({ type: 'attest-alignment', actorUserId: chooseCauseAttester(activeUsers, project.causeId, random).id, causeId: project.causeId, projectId: project.id, statementId, alignment: 'supports-described-outcome', dependsOn: [createProjects.get(project.id)!.id, publishes.get(statementId)!.id] })); } + const fundableProjects = projects.slice(0, Math.max(1, Math.floor(projects.length * 0.8))); for (let index = 0; index < countByType['fund-project']; index++) { - const project = random.pick(projects); const supporters = activeUsers.filter((user) => user.causeIds.includes(project.causeId) && user.fundingWeight > 0); const actor = random.weighted(supporters, (user) => user.fundingWeight); + const project = random.weighted(fundableProjects, (item) => Math.max(1, fundableProjects.length - projects.indexOf(item))); const supporters = activeUsers.filter((user) => user.causeIds.includes(project.causeId) && user.fundingWeight > 0); const actor = random.weighted(supporters, (user) => user.fundingWeight); const alignment = alignments.find((item) => item.projectId === project.id); if (!alignment) throw new Error(`impossible fund-project: ${project.id} has no alignment action`); - add({ type: 'fund-project', actorUserId: actor.id, causeId: project.causeId, projectId: project.id, dependsOn: [createProjects.get(project.id)!.id, alignment.id] }); + add({ type: 'fund-project', actorUserId: actor.id, causeId: project.causeId, projectId: project.id, amount: fundingAmount(actor, projects.indexOf(project), random), dependsOn: [createProjects.get(project.id)!.id, alignment.id] }); } const deposits: PlannedAction[] = []; - for (let index = 0; index < countByType['deposit-note']; index++) { const actor = random.weighted(activeUsers, (user) => user.activityWeight); deposits.push(add({ type: 'deposit-note', actorUserId: actor.id, noteId: `note-${String(index + 1).padStart(4, '0')}`, dependsOn: [] })); } + const delegatingDepositors = activeUsers.filter((owner) => delegates.some((delegate) => delegate.id !== owner.id && delegate.causeIds.some((causeId) => owner.causeIds.includes(causeId)))); + if (delegatingDepositors.length === 0) throw new Error('campaign has no users connected to the delegate trust graph'); + for (let index = 0; index < countByType['deposit-note']; index++) { const actor = random.weighted(delegatingDepositors, (user) => user.activityWeight); deposits.push(add({ type: 'deposit-note', actorUserId: actor.id, noteId: `note-${String(index + 1).padStart(4, '0')}`, amount: Math.max(100, Math.round(actor.fundingWeight * 500)) * random.integer(1, 3), dependsOn: [] })); } const delegations: PlannedAction[] = []; - for (let index = 0; index < countByType['delegate-note']; index++) { const deposit = deposits[index % deposits.length]; const delegate = random.pick(delegates.filter((user) => user.id !== deposit.actorUserId)); delegations.push(add({ type: 'delegate-note', actorUserId: deposit.actorUserId, noteId: deposit.noteId, delegateUserId: delegate.id, dependsOn: [deposit.id] })); } + for (let index = 0; index < countByType['delegate-note']; index++) { + const deposit = deposits[index % deposits.length]; const owner = users.find((user) => user.id === deposit.actorUserId)!; + const candidates = delegates.filter((user) => user.id !== owner.id && user.causeIds.some((causeId) => owner.causeIds.includes(causeId))); + if (candidates.length === 0) throw new Error(`no cause-aware delegate available for ${owner.id}`); + const delegate = random.weighted(candidates, (user) => user.activityWeight * user.causeIds.filter((causeId) => owner.causeIds.includes(causeId)).length); + const sharedCauseIds = delegate.causeIds.filter((causeId) => owner.causeIds.includes(causeId)).sort(); + delegations.push(add({ type: 'delegate-note', actorUserId: owner.id, noteId: deposit.noteId, delegateUserId: delegate.id, amount: deposit.amount, delegationBasis: { sharedCauseIds, reason: 'shared-cause-trusted-role' }, dependsOn: [deposit.id] })); + } for (let index = 0; index < countByType['revoke-delegation']; index++) { const delegation = delegations[index % delegations.length]; add({ type: 'revoke-delegation', actorUserId: delegation.actorUserId, noteId: delegation.noteId, dependsOn: [delegation.id] }); } validatePlannedActions(manifest, statements, users, projects, actions); const writesByType = Object.fromEntries(manifest.actionRules.map((rule) => [rule.type, actions.filter((action) => action.type === rule.type).length])) as Record; const estimatedGasByType = Object.fromEntries(Object.entries(writesByType).map(([type, count]) => [type, count * GAS_UNITS[type as CampaignActionType]])) as Record; - return { version: CAMPAIGN_PLAN_VERSION, campaignId: manifest.campaign.id, deterministicSeed: manifest.campaign.deterministicSeed, manifestFingerprint: sha256(stableJson(manifest)), statements, users, projects, actions, estimate: { writesByType, totalWrites: actions.length, estimatedGasByType, estimatedTotalGas: Object.values(estimatedGasByType).reduce((sum, value) => sum + value, 0), assumptions: { gasUnitsPerWrite: GAS_UNITS, paymentTokenUnitsPerFunding: 100 }, estimatedPaymentTokenUnits: writesByType['fund-project'] * 100 } }; + const estimatedPaymentTokenUnits = actions.filter((action) => action.type === 'fund-project').reduce((sum, action) => sum + (action.amount ?? 0), 0); + return { version: CAMPAIGN_PLAN_VERSION, campaignId: manifest.campaign.id, deterministicSeed: manifest.campaign.deterministicSeed, manifestFingerprint: sha256(stableJson(manifest)), statements, users, projects, actions, estimate: { writesByType, totalWrites: actions.length, estimatedGasByType, estimatedTotalGas: Object.values(estimatedGasByType).reduce((sum, value) => sum + value, 0), assumptions: { gasUnitsPerWrite: GAS_UNITS, paymentTokenBaseUnit: 100 }, estimatedPaymentTokenUnits } }; } export function validatePlannedActions(manifest: CampaignManifestV1, statements: PlannedStatement[], users: PlannedUser[], projects: PlannedProject[], actions: PlannedAction[]): void { @@ -230,6 +274,12 @@ export function validatePlannedActions(manifest: CampaignManifestV1, statements: for (const dependencyId of action.dependsOn) { const dependency = actionById.get(dependencyId); if (!dependency || dependency.sequence >= action.sequence) throw new Error(`${action.id} has impossible dependency ${dependencyId}`); } const rule = manifest.actionRules.find((item) => item.type === action.type)!; for (const prerequisite of rule.prerequisites) if (!action.dependsOn.some((id) => actionById.get(id)?.type === prerequisite)) throw new Error(`${action.id} is missing ${prerequisite} prerequisite`); + const actor = action.actorUserId ? users.find((user) => user.id === action.actorUserId) : undefined; + if (action.causeId && actor && ['set-belief', 'fund-project'].includes(action.type) && !actor.causeIds.includes(action.causeId)) throw new Error(`${action.id} actor is outside cause ${action.causeId}`); + if (action.type === 'set-belief' && !action.belief) throw new Error(`${action.id} is missing belief value`); + if (action.type === 'fund-project' && (!action.amount || action.amount <= 0)) throw new Error(`${action.id} is missing positive funding amount`); + if (action.type === 'attest-alignment' && action.alignment !== 'supports-described-outcome') throw new Error(`${action.id} lacks project outcome alignment evidence`); + if (action.type === 'delegate-note' && (!action.delegationBasis || action.delegationBasis.sharedCauseIds.length === 0)) throw new Error(`${action.id} lacks a shared-cause delegation basis`); } for (const rule of manifest.actionRules) { const count = actions.filter((action) => action.type === rule.type).length; if (count < rule.targetCount.min || count > rule.targetCount.max) throw new Error(`${rule.type} planned count ${count} is outside target ${rule.targetCount.min}-${rule.targetCount.max}`); } if (users.some((user) => user.causeIds.length < 1 || user.causeIds.length > 3)) throw new Error('user cause assignment is outside 1-3 causes'); diff --git a/fake-data-generation/test/campaignPlanner.test.ts b/fake-data-generation/test/campaignPlanner.test.ts index d4551492..78aef30f 100644 --- a/fake-data-generation/test/campaignPlanner.test.ts +++ b/fake-data-generation/test/campaignPlanner.test.ts @@ -31,11 +31,56 @@ test('all implication actions use accepted bridge-role pairs', async () => { const plan = await buildCampaignPlan(await loadManifest()); const statements = new Map(plan.statements.map((statement) => [statement.id, statement])); for (const action of plan.actions.filter((item) => item.type === 'attest-implication')) { assert.equal(action.implication?.evidence, 'accepted-bridge-role-pair'); + assert.ok(plan.users.find((user) => user.id === action.actorUserId)!.causeIds.includes(action.causeId!)); assert.match(statements.get(action.implication!.fromStatementId)!.role!, /^modified-(left|right)$/); assert.equal(statements.get(action.implication!.toStatementId)!.role, 'commonality'); } }); +test('behavior histories are cause-aware and carry executable intent', async () => { + const plan = await buildCampaignPlan(await loadManifest()); + const users = new Map(plan.users.map((user) => [user.id, user])); + const projects = new Map(plan.projects.map((project) => [project.id, project])); + const actions = new Map(plan.actions.map((action) => [action.id, action])); + + for (const action of plan.actions.filter((item) => item.type === 'set-belief')) { + assert.ok(users.get(action.actorUserId!)!.causeIds.includes(action.causeId!)); + assert.ok(action.belief === 'believe' || action.belief === 'disbelieve'); + } + assert.ok(plan.actions.some((action) => action.type === 'set-belief' && action.dependsOn.some((id) => actions.get(id)?.type === 'set-belief')), 'expected belief changes'); + + for (const action of plan.actions.filter((item) => item.type === 'attest-alignment')) { + assert.ok(users.get(action.actorUserId!)!.causeIds.includes(action.causeId!)); + assert.ok(projects.get(action.projectId!)!.statementIds.includes(action.statementId!)); + assert.equal(action.alignment, 'supports-described-outcome'); + } + assert.ok(plan.projects.every((project) => project.title.length > 20 && project.outcome.length > 20)); + for (const action of plan.actions.filter((item) => item.type === 'fund-project')) { + assert.ok(users.get(action.actorUserId!)!.causeIds.includes(action.causeId!)); + assert.ok(action.amount! > 0); + } + const fundingCounts = plan.projects.map((project) => plan.actions.filter((action) => action.type === 'fund-project' && action.projectId === project.id).length); + assert.ok(fundingCounts.some((count) => count === 0), 'expected deliberately unfunded projects'); + assert.ok(Math.max(...fundingCounts) >= Math.max(10, Math.min(...fundingCounts.filter((count) => count > 0)) * 3), 'expected skewed project popularity'); + assert.ok(new Set(plan.actions.filter((item) => item.type === 'fund-project').map((item) => item.amount)).size > 10, 'expected uneven funding amounts'); + assert.equal(plan.estimate.estimatedPaymentTokenUnits, plan.actions.filter((item) => item.type === 'fund-project').reduce((sum, item) => sum + item.amount!, 0)); +}); + +test('delegations follow a shared-cause trust graph and revocations follow delegations', async () => { + const plan = await buildCampaignPlan(await loadManifest()); + const users = new Map(plan.users.map((user) => [user.id, user])); + const actions = new Map(plan.actions.map((action) => [action.id, action])); + for (const action of plan.actions.filter((item) => item.type === 'delegate-note')) { + const owner = users.get(action.actorUserId!)!; const delegate = users.get(action.delegateUserId!)!; + assert.ok(delegate.roles.includes('delegate')); + assert.deepEqual(action.delegationBasis!.sharedCauseIds, delegate.causeIds.filter((causeId) => owner.causeIds.includes(causeId)).sort()); + assert.ok(action.amount! > 0); + } + for (const action of plan.actions.filter((item) => item.type === 'revoke-delegation')) { + assert.ok(action.dependsOn.some((id) => actions.get(id)?.type === 'delegate-note')); + } +}); + test('validation rejects an action whose prerequisite points forward', async () => { const manifest = await loadManifest(); const plan = await buildCampaignPlan(manifest); const invalid = structuredClone(plan.actions); invalid[0].dependsOn = [invalid[1].id]; From 96a07008502418f3c0401baa54954ba717b5e8f6 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 14:12:19 -0400 Subject: [PATCH 07/19] Add campaign reconciliation engine --- fake-data-generation/README.md | 2 + .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignExecutor.ts | 7 ++ fake-data-generation/campaignReconciler.ts | 70 +++++++++++++++++++ .../test/campaignReconciler.test.ts | 38 ++++++++++ 5 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 fake-data-generation/campaignReconciler.ts create mode 100644 fake-data-generation/test/campaignReconciler.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 379c165c..f04848ac 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -17,6 +17,8 @@ Campaign execution uses the explicit environment boundary in `campaignEnvironmen The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. No CLI or real contract adapter is exposed yet: this layer is deliberately tested against a local fake chain before the campaign behavior and contract bindings are added. +`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. A concrete contract/indexer adapter supplies raw entity matches and SDK-derived checks for every mined action; the reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. + ## Overview The generative testing suite consists of: diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 872aca48..2f8539a1 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -182,7 +182,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. -6. **[ ] Build transaction-to-indexer deep reconciliation.** Cover every selected action type, derived totals, duplicates, and settling-window behavior. Produce machine-readable results plus a concise human report. +6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract now cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. Next bind all selected action types to the real indexer and SDK, then exercise those mappings against the deep local stack. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. diff --git a/fake-data-generation/campaignExecutor.ts b/fake-data-generation/campaignExecutor.ts index 57256c29..f943cea5 100644 --- a/fake-data-generation/campaignExecutor.ts +++ b/fake-data-generation/campaignExecutor.ts @@ -11,6 +11,7 @@ export interface CampaignReceipt { status: 'success' | 'reverted'; gasUsed: bigint; effectiveGasPrice: bigint; + blockNumber?: bigint; } export interface CampaignExecutionAdapter { @@ -25,6 +26,9 @@ export interface CampaignActionExecution { status: CampaignActionStatus; attempts: number; transactionHash?: Hex; + submittedAt?: string; + minedAt?: string; + blockNumber?: string; gasUsed?: string; nativeCost?: string; failure?: { category: string; message: string }; @@ -139,6 +143,7 @@ export async function executeCampaignPlan(input: { const transactionHash = await adapter.submit(action); lastSubmissionAt = Date.now(); record.status = 'submitted'; record.transactionHash = transactionHash; record.nativeCost = estimate.toString(); + record.submittedAt = now().toISOString(); transactionCount += 1; reservedNativeCost += estimate; await save(); } finally { release(); } @@ -157,6 +162,8 @@ export async function executeCampaignPlan(input: { const actualCost = receipt.gasUsed * receipt.effectiveGasPrice; reservedNativeCost += actualCost - previousCost; record.gasUsed = receipt.gasUsed.toString(); record.nativeCost = actualCost.toString(); + record.minedAt = now().toISOString(); + if (receipt.blockNumber !== undefined) record.blockNumber = receipt.blockNumber.toString(); record.status = receipt.status === 'success' ? 'mined' : 'failed'; if (receipt.status === 'reverted') record.failure = { category: 'contract-revert', message: 'transaction reverted' }; await save(); diff --git a/fake-data-generation/campaignReconciler.ts b/fake-data-generation/campaignReconciler.ts new file mode 100644 index 00000000..483e795d --- /dev/null +++ b/fake-data-generation/campaignReconciler.ts @@ -0,0 +1,70 @@ +import { mkdir, rename, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import type { Hex } from 'viem'; +import type { CampaignActionType } from './campaignSchema.js'; +import type { CampaignExecutionState } from './campaignExecutor.js'; +import type { PlannedAction } from './campaignPlanner.js'; + +export const CAMPAIGN_RECONCILIATION_VERSION = 'commonality-campaign-reconciliation-v1' as const; +export interface IndexedActionMatch { entityId: string; eventId: string; transactionHash: Hex; indexedAt?: string } +export interface DerivedCheck { name: string; expected: string | number | boolean | null; actual: string | number | boolean | null } +export interface CampaignReconciliationAdapter { + getChainHead(): Promise; + getIndexerHead(): Promise; + findIndexedAction(action: PlannedAction, transactionHash: Hex): Promise; + getDerivedChecks(action: PlannedAction): Promise; +} +export type ReconciliationStatus = 'verified' | 'pending' | 'missing' | 'duplicate' | 'derived-mismatch' | 'not-mined'; +export interface ActionReconciliation { actionId: string; type: CampaignActionType; status: ReconciliationStatus; transactionHash?: Hex; indexedMatches: IndexedActionMatch[]; derivedChecks: DerivedCheck[]; indexLatencyMs?: number } +export interface CampaignReconciliationReport { + version: typeof CAMPAIGN_RECONCILIATION_VERSION; campaignId: string; manifestFingerprint: string; generatedAt: string; + settlingWindowMs: number; settled: boolean; chainHead: string; indexerHead: string; chainHeadLag: string; + countsByStatus: Record; countsByType: Record>; actions: ActionReconciliation[]; +} +export interface CampaignReconciliationOptions { settlingWindowMs: number; pollIntervalMs: number; now?: () => Date; sleep?: (milliseconds: number) => Promise } + +const statuses: ReconciliationStatus[] = ['verified', 'pending', 'missing', 'duplicate', 'derived-mismatch', 'not-mined']; +const emptyCounts = (): Record => Object.fromEntries(statuses.map((status) => [status, 0])) as Record; +const delay = (milliseconds: number): Promise => new Promise((resolve) => setTimeout(resolve, milliseconds)); + +export async function reconcileCampaign(input: { actions: readonly PlannedAction[]; execution: CampaignExecutionState; adapter: CampaignReconciliationAdapter; options: CampaignReconciliationOptions }): Promise { + const { actions, execution, adapter, options } = input; + if (options.settlingWindowMs < 0 || options.pollIntervalMs < 0) throw new Error('settling and poll intervals must be non-negative'); + if (actions.length === 0) throw new Error('cannot reconcile an empty campaign plan'); + if (execution.actions.length !== actions.length || execution.actions.some((record, index) => record.actionId !== actions[index]?.id)) throw new Error('execution state action list does not match reconciliation plan'); + const now = options.now ?? (() => new Date()); const sleep = options.sleep ?? delay; const started = now().getTime(); + let observations: ActionReconciliation[] = []; let chainHead = 0n; let indexerHead = 0n; + for (;;) { + [chainHead, indexerHead] = await Promise.all([adapter.getChainHead(), adapter.getIndexerHead()]); + observations = await Promise.all(actions.map(async (action, index): Promise => { + const record = execution.actions[index]; + if (record.status !== 'mined' || !record.transactionHash) return { actionId: action.id, type: action.type, status: 'not-mined', indexedMatches: [], derivedChecks: [] }; + const [indexedMatches, derivedChecks] = await Promise.all([adapter.findIndexedAction(action, record.transactionHash), adapter.getDerivedChecks(action)]); + let status: ReconciliationStatus = 'verified'; + if (indexedMatches.length === 0) status = 'pending'; + else if (indexedMatches.length > 1 || new Set(indexedMatches.map((match) => match.entityId)).size > 1) status = 'duplicate'; + else if (derivedChecks.some((check) => check.actual !== check.expected)) status = 'derived-mismatch'; + const indexedAt = indexedMatches[0]?.indexedAt ? Date.parse(indexedMatches[0].indexedAt) : undefined; + const minedAt = record.minedAt ? Date.parse(record.minedAt) : undefined; + return { actionId: action.id, type: action.type, status, transactionHash: record.transactionHash, indexedMatches, derivedChecks, ...(indexedAt !== undefined && minedAt !== undefined ? { indexLatencyMs: Math.max(0, indexedAt - minedAt) } : {}) }; + })); + if (!observations.some((item) => item.status === 'pending') || now().getTime() - started >= options.settlingWindowMs) break; + await sleep(options.pollIntervalMs); + } + if (now().getTime() - started >= options.settlingWindowMs) observations = observations.map((item) => item.status === 'pending' ? { ...item, status: 'missing' } : item); + const countsByStatus = emptyCounts(); const countsByType: Record> = {}; + for (const item of observations) { countsByStatus[item.status] += 1; (countsByType[item.type] ??= emptyCounts())[item.status] += 1; } + return { version: CAMPAIGN_RECONCILIATION_VERSION, campaignId: execution.campaignId, manifestFingerprint: execution.manifestFingerprint, generatedAt: now().toISOString(), settlingWindowMs: options.settlingWindowMs, settled: !observations.some((item) => item.status === 'pending'), chainHead: chainHead.toString(), indexerHead: indexerHead.toString(), chainHeadLag: (chainHead > indexerHead ? chainHead - indexerHead : 0n).toString(), countsByStatus, countsByType, actions: observations }; +} + +export function formatReconciliationReport(report: CampaignReconciliationReport): string { + const failures = report.actions.filter((action) => !['verified', 'not-mined'].includes(action.status)); + return [`Campaign ${report.campaignId}: ${report.countsByStatus.verified}/${report.actions.length} actions verified`, `Indexer head ${report.indexerHead}; chain head ${report.chainHead}; lag ${report.chainHeadLag} blocks`, failures.length === 0 ? 'No indexing or derived-state discrepancies.' : `Discrepancies: ${failures.map((item) => `${item.actionId} (${item.status})`).join(', ')}`].join('\n'); +} + +export async function writeReconciliationArtifacts(report: CampaignReconciliationReport, jsonPath: string, summaryPath: string): Promise { + await mkdir(path.dirname(jsonPath), { recursive: true }); await mkdir(path.dirname(summaryPath), { recursive: true }); + const jsonTemporary = `${jsonPath}.tmp`; const summaryTemporary = `${summaryPath}.tmp`; + await writeFile(jsonTemporary, `${JSON.stringify(report, null, 2)}\n`); await writeFile(summaryTemporary, `${formatReconciliationReport(report)}\n`); + await rename(jsonTemporary, jsonPath); await rename(summaryTemporary, summaryPath); +} diff --git a/fake-data-generation/test/campaignReconciler.test.ts b/fake-data-generation/test/campaignReconciler.test.ts new file mode 100644 index 00000000..6191bc37 --- /dev/null +++ b/fake-data-generation/test/campaignReconciler.test.ts @@ -0,0 +1,38 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import type { Hex } from 'viem'; +import { formatReconciliationReport, reconcileCampaign, type CampaignReconciliationAdapter } from '../campaignReconciler.js'; +import { CAMPAIGN_EXECUTION_VERSION, type CampaignExecutionState } from '../campaignExecutor.js'; +import type { PlannedAction } from '../campaignPlanner.js'; + +const hash = (value: number): Hex => `0x${value.toString(16).padStart(64, '0')}` as Hex; +const actions: PlannedAction[] = [ + { id: 'a1', sequence: 1, type: 'publish-statement', actorUserId: null, statementId: 's1', dependsOn: [] }, + { id: 'a2', sequence: 2, type: 'set-belief', actorUserId: 'u1', statementId: 's1', belief: 'believe', dependsOn: ['a1'] }, + { id: 'a3', sequence: 3, type: 'fund-project', actorUserId: 'u1', projectId: 'p1', amount: 10, dependsOn: [] }, + { id: 'a4', sequence: 4, type: 'deposit-note', actorUserId: 'u1', noteId: 'n1', amount: 10, dependsOn: [] }, +]; +const execution: CampaignExecutionState = { version: CAMPAIGN_EXECUTION_VERSION, campaignId: 'campaign', manifestFingerprint: 'fp', updatedAt: '2026-01-01T00:00:00.000Z', actions: actions.map((action, index) => ({ actionId: action.id, status: index === 3 ? 'failed' : 'mined', attempts: 1, transactionHash: index === 3 ? undefined : hash(index + 1), minedAt: '2026-01-01T00:00:00.000Z' })) }; + +test('classifies indexed, duplicate, derived mismatch, and unmined actions', async () => { + const adapter: CampaignReconciliationAdapter = { + getChainHead: async () => 105n, getIndexerHead: async () => 103n, + findIndexedAction: async (action, transactionHash) => action.id === 'a2' ? [{ entityId: 'one', eventId: '1', transactionHash }, { entityId: 'two', eventId: '2', transactionHash }] : [{ entityId: action.id, eventId: action.id, transactionHash, indexedAt: '2026-01-01T00:00:01.000Z' }], + getDerivedChecks: async (action) => action.id === 'a3' ? [{ name: 'project funding', expected: 10, actual: 9 }] : [{ name: 'visible', expected: true, actual: true }], + }; + const report = await reconcileCampaign({ actions, execution, adapter, options: { settlingWindowMs: 0, pollIntervalMs: 0 } }); + assert.deepEqual(report.actions.map((item) => item.status), ['verified', 'duplicate', 'derived-mismatch', 'not-mined']); + assert.equal(report.actions[0].indexLatencyMs, 1000); assert.equal(report.chainHeadLag, '2'); + assert.match(formatReconciliationReport(report), /a2 \(duplicate\), a3 \(derived-mismatch\)/); +}); + +test('polls through lag and marks omissions missing after the settling window', async () => { + let time = 0; let reads = 0; + const adapter: CampaignReconciliationAdapter = { + getChainHead: async () => 10n, getIndexerHead: async () => 9n, + findIndexedAction: async (action, transactionHash) => { reads += 1; return action.id === 'a1' && reads >= 4 ? [{ entityId: 's1', eventId: 'e1', transactionHash }] : []; }, + getDerivedChecks: async () => [], + }; + const report = await reconcileCampaign({ actions: actions.slice(0, 2), execution: { ...execution, actions: execution.actions.slice(0, 2) }, adapter, options: { settlingWindowMs: 20, pollIntervalMs: 10, now: () => new Date(time), sleep: async (ms) => { time += ms; } } }); + assert.equal(report.actions[0].status, 'verified'); assert.equal(report.actions[1].status, 'missing'); assert.equal(report.settled, true); +}); From d5672cc6f5d3c847c2a59bc0a6e1074ff138b970 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 14:21:51 -0400 Subject: [PATCH 08/19] Bind campaign reconciliation to indexer events --- fake-data-generation/README.md | 2 +- .../TESTNET-SIMULATION-PLAN.md | 2 +- .../campaignIndexerAdapter.ts | 77 +++++++++++++++++++ .../test/campaignIndexerAdapter.test.ts | 58 ++++++++++++++ 4 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 fake-data-generation/campaignIndexerAdapter.ts create mode 100644 fake-data-generation/test/campaignIndexerAdapter.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index f04848ac..46b63163 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -17,7 +17,7 @@ Campaign execution uses the explicit environment boundary in `campaignEnvironmen The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. No CLI or real contract adapter is exposed yet: this layer is deliberately tested against a local fake chain before the campaign behavior and contract bindings are added. -`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. A concrete contract/indexer adapter supplies raw entity matches and SDK-derived checks for every mined action; the reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. +`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to an explicit SDK-check provider. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. The remaining binding work is to define the runtime ID/address/CID artifact produced by execution and implement the SDK-check provider over it before exercising the deep local stack. ## Overview diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 2f8539a1..44153f82 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -182,7 +182,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. -6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract now cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. Next bind all selected action types to the real indexer and SDK, then exercise those mappings against the deep local stack. +6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter now maps every selected action type to its real Ponder event(s), filters by mined transaction, reads chain/indexer heads, and exposes an explicit SDK-derived-check seam. Next define the execution-produced runtime ID/address/CID bindings and implement the SDK checks over them, then exercise both layers against the deep local stack. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. diff --git a/fake-data-generation/campaignIndexerAdapter.ts b/fake-data-generation/campaignIndexerAdapter.ts new file mode 100644 index 00000000..90d7ac56 --- /dev/null +++ b/fake-data-generation/campaignIndexerAdapter.ts @@ -0,0 +1,77 @@ +import type { PublicClient } from 'viem'; +import type { SDKMachinery } from '@commonality/sdk/machinery'; +import { chainStatusKeyForChainId, fetchEvents, type RawEventFromCache } from '@commonality/sdk/utils'; +import type { CampaignActionType } from './campaignSchema.js'; +import type { PlannedAction } from './campaignPlanner.js'; +import type { CampaignReconciliationAdapter, DerivedCheck, IndexedActionMatch } from './campaignReconciler.js'; + +/** Events whose presence proves that one planned write reached the raw event cache. */ +export const CAMPAIGN_ACTION_EVENTS: Record = { + 'publish-statement': ['DataPublished'], + 'create-cause': ['RefUpdated'], + 'set-belief': ['DirectSupport'], + 'attest-implication': ['ImplicationAttestation'], + 'create-project': ['ProjectCreated'], + 'attest-alignment': ['AlignmentAttestation'], + // The campaign contract adapter may use either ordinary crowdfunding or a + // retroactive donation. Both are user-visible funding writes in SDK folds. + 'fund-project': ['ERC1155Bought', 'RetroactiveDonationReceived'], + 'deposit-note': ['NoteCreated'], + 'delegate-note': ['NoteDelegated'], + 'revoke-delegation': ['NoteRevoked'], +}; + +export interface CampaignDerivedCheckProvider { + /** Run the SDK fold checks appropriate to this action and its runtime bindings. */ + getDerivedChecks(action: PlannedAction): Promise; +} + +type StatusResult = Record; + +function requireEventCacheUrl(machinery: SDKMachinery): string { + if (!machinery.eventCacheUrl) throw new Error('campaign reconciliation requires an event-cache URL'); + return machinery.eventCacheUrl; +} + +async function getIndexerHead(machinery: SDKMachinery): Promise { + const eventCacheUrl = requireEventCacheUrl(machinery); + const response = await fetch(`${new URL(eventCacheUrl).origin}/status`); + if (!response.ok) throw new Error(`indexer status endpoint returned ${response.status}`); + const status = await response.json() as StatusResult; + const key = machinery.chainStatusKey ?? chainStatusKeyForChainId(machinery.defaultChainId ?? 31_337); + const block = status[key]?.block?.number; + if (!Number.isSafeInteger(block) || block! < 0) throw new Error(`indexer status has no valid ${key} block`); + return BigInt(block!); +} + +function toMatch(event: RawEventFromCache): IndexedActionMatch { + return { + entityId: event.id, + eventId: `${event.eventName}:${event.id}`, + transactionHash: event.transactionHash as `0x${string}`, + }; +} + +/** + * Bind campaign reconciliation to the real Ponder event cache and SDK fold seam. + * Event-cache results are filtered by transaction hash because the public API + * deliberately exposes no transactionHash query parameter. + */ +export function createCampaignIndexerAdapter(input: { + machinery: SDKMachinery; + publicClient: Pick; + derivedChecks: CampaignDerivedCheckProvider; +}): CampaignReconciliationAdapter { + requireEventCacheUrl(input.machinery); + return { + getChainHead: () => input.publicClient.getBlockNumber(), + getIndexerHead: () => getIndexerHead(input.machinery), + async findIndexedAction(action, transactionHash) { + const eventGroups = await Promise.all(CAMPAIGN_ACTION_EVENTS[action.type].map((eventName) => + fetchEvents(input.machinery, { eventName, limit: 1000 }))); + const target = transactionHash.toLowerCase(); + return eventGroups.flat().filter((event) => event.transactionHash.toLowerCase() === target).map(toMatch); + }, + getDerivedChecks: (action) => input.derivedChecks.getDerivedChecks(action), + }; +} diff --git a/fake-data-generation/test/campaignIndexerAdapter.test.ts b/fake-data-generation/test/campaignIndexerAdapter.test.ts new file mode 100644 index 00000000..81ef3474 --- /dev/null +++ b/fake-data-generation/test/campaignIndexerAdapter.test.ts @@ -0,0 +1,58 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { createCampaignIndexerAdapter, CAMPAIGN_ACTION_EVENTS } from '../campaignIndexerAdapter.js'; +import type { PlannedAction } from '../campaignPlanner.js'; + +const TX = `0x${'a'.repeat(64)}` as const; +const OTHER_TX = `0x${'b'.repeat(64)}` as const; +const action: PlannedAction = { id: 'a1', sequence: 1, type: 'fund-project', actorUserId: 'u1', projectId: 'p1', amount: 10, dependsOn: [] }; + +test('every campaign action has an explicit real indexer event mapping', () => { + assert.deepEqual(Object.keys(CAMPAIGN_ACTION_EVENTS).sort(), [ + 'attest-alignment', 'attest-implication', 'create-cause', 'create-project', 'delegate-note', + 'deposit-note', 'fund-project', 'publish-statement', 'revoke-delegation', 'set-belief', + ]); + assert.ok(Object.values(CAMPAIGN_ACTION_EVENTS).every((names) => names.length > 0)); +}); + +test('adapter reads chain/indexer heads and matches only the action transaction', async () => { + const originalFetch = globalThis.fetch; + const requested: string[] = []; + globalThis.fetch = (async (request: string | URL | Request) => { + const url = String(request); requested.push(url); + if (url === 'http://indexer.test/status') return new Response(JSON.stringify({ anvil: { block: { number: 41 } } })); + const eventName = new URL(url).searchParams.get('eventName')!; + return new Response(JSON.stringify({ items: [ + { id: `${eventName}-wanted`, contractAddress: '0x1', eventName, blockNumber: '40', blockTimestamp: '1', transactionHash: TX, logIndex: 0, topic0: null, topic1: null, topic2: null, topic3: null, data: '0x' }, + { id: `${eventName}-other`, contractAddress: '0x1', eventName, blockNumber: '40', blockTimestamp: '1', transactionHash: OTHER_TX, logIndex: 1, topic0: null, topic1: null, topic2: null, topic3: null, data: '0x' }, + ] })); + }) as typeof fetch; + try { + const adapter = createCampaignIndexerAdapter({ + machinery: { ipfsConfig: {}, twitterApiConfig: {}, testConfig: {}, eventCacheUrl: 'http://indexer.test/api', defaultChainId: 31337, chainStatusKey: 'anvil' }, + publicClient: { getBlockNumber: async () => 43n } as never, + derivedChecks: { getDerivedChecks: async () => [{ name: 'SDK project funding', expected: 10, actual: 10 }] }, + }); + assert.equal(await adapter.getChainHead(), 43n); + assert.equal(await adapter.getIndexerHead(), 41n); + const matches = await adapter.findIndexedAction(action, TX); + assert.equal(matches.length, 2); + assert.deepEqual(matches.map((match) => match.eventId).sort(), ['ERC1155Bought:ERC1155Bought-wanted', 'RetroactiveDonationReceived:RetroactiveDonationReceived-wanted']); + assert.deepEqual(await adapter.getDerivedChecks(action), [{ name: 'SDK project funding', expected: 10, actual: 10 }]); + assert.ok(requested.some((url) => url.includes('eventName=ERC1155Bought'))); + assert.ok(requested.some((url) => url.includes('eventName=RetroactiveDonationReceived'))); + } finally { globalThis.fetch = originalFetch; } +}); + +test('adapter fails closed when indexer status omits the configured chain', async () => { + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => new Response(JSON.stringify({}))) as typeof fetch; + try { + const adapter = createCampaignIndexerAdapter({ + machinery: { ipfsConfig: {}, twitterApiConfig: {}, testConfig: {}, eventCacheUrl: 'http://indexer.test/api', chainStatusKey: 'base-sepolia' }, + publicClient: { getBlockNumber: async () => 1n } as never, + derivedChecks: { getDerivedChecks: async () => [] }, + }); + await assert.rejects(adapter.getIndexerHead(), /no valid base-sepolia block/); + } finally { globalThis.fetch = originalFetch; } +}); From 729d18b5a8f433e07f2e30902131df99e52aeb65 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 14:27:15 -0400 Subject: [PATCH 09/19] Add campaign runtime binding artifact --- fake-data-generation/README.md | 2 +- .../TESTNET-SIMULATION-PLAN.md | 2 +- .../campaignRuntimeBindings.ts | 121 ++++++++++++++++++ .../test/campaignRuntimeBindings.test.ts | 55 ++++++++ 4 files changed, 178 insertions(+), 2 deletions(-) create mode 100644 fake-data-generation/campaignRuntimeBindings.ts create mode 100644 fake-data-generation/test/campaignRuntimeBindings.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 46b63163..7141ec0a 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -17,7 +17,7 @@ Campaign execution uses the explicit environment boundary in `campaignEnvironmen The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. No CLI or real contract adapter is exposed yet: this layer is deliberately tested against a local fake chain before the campaign behavior and contract bindings are added. -`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to an explicit SDK-check provider. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. The remaining binding work is to define the runtime ID/address/CID artifact produced by execution and implement the SDK-check provider over it before exercising the deep local stack. +`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to an explicit SDK-check provider. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. The remaining reconciliation work is to implement the SDK-check provider over that artifact before exercising the deep local stack. ## Overview diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 44153f82..6f016232 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -182,7 +182,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. -6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter now maps every selected action type to its real Ponder event(s), filters by mined transaction, reads chain/indexer heads, and exposes an explicit SDK-derived-check seam. Next define the execution-produced runtime ID/address/CID bindings and implement the SDK checks over them, then exercise both layers against the deep local stack. +6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter maps every selected action type to its real Ponder event(s), filters by mined transaction, reads chain/indexer heads, and exposes an explicit SDK-derived-check seam. The validated, atomically persisted runtime-binding artifact now maps planned IDs to execution-produced wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs without containing secrets. Next implement the SDK checks over those bindings, then exercise both layers against the deep local stack. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. diff --git a/fake-data-generation/campaignRuntimeBindings.ts b/fake-data-generation/campaignRuntimeBindings.ts new file mode 100644 index 00000000..bf1730b5 --- /dev/null +++ b/fake-data-generation/campaignRuntimeBindings.ts @@ -0,0 +1,121 @@ +import { mkdir, readFile, rename, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { getAddress, isAddress, zeroAddress, type Address } from 'viem'; +import { ensureIpfsCidV1, type IpfsCidV1 } from '@commonality/sdk/utils'; +import type { CampaignPlan } from './campaignPlanner.js'; + +export const CAMPAIGN_RUNTIME_BINDINGS_VERSION = 'commonality-campaign-runtime-bindings-v1' as const; + +export interface CampaignCauseBinding { + owner: Address; + refName: string; + rosterCid: IpfsCidV1; +} + +export interface CampaignNoteBinding { + contractAddress: Address; + noteId: string; +} + +/** + * Public, execution-produced identifiers needed to turn planned IDs into SDK + * queries. This artifact deliberately contains no wallet private keys. + */ +export interface CampaignRuntimeBindings { + version: typeof CAMPAIGN_RUNTIME_BINDINGS_VERSION; + campaignId: string; + manifestFingerprint: string; + updatedAt: string; + users: Record; + statements: Record; + causes: Record; + projects: Record; + notes: Record; +} + +export function createEmptyRuntimeBindings(plan: CampaignPlan, now: Date = new Date()): CampaignRuntimeBindings { + return { + version: CAMPAIGN_RUNTIME_BINDINGS_VERSION, + campaignId: plan.campaignId, + manifestFingerprint: plan.manifestFingerprint, + updatedAt: now.toISOString(), + users: {}, statements: {}, causes: {}, projects: {}, notes: {}, + }; +} + +function validateAddress(value: unknown, label: string): asserts value is Address { + if (typeof value !== 'string' || !isAddress(value) || getAddress(value) === zeroAddress) { + throw new Error(`${label} is not a valid non-zero address`); + } +} + +function validateKeys(label: string, values: Record, allowed: Set): void { + for (const id of Object.keys(values)) if (!allowed.has(id)) throw new Error(`${label} contains unknown planned ID ${id}`); +} + +function validateBindingValues(bindings: CampaignRuntimeBindings): void { + for (const [id, address] of Object.entries(bindings.users)) validateAddress(address, `user ${id}`); + for (const [id, cid] of Object.entries(bindings.statements)) { + try { ensureIpfsCidV1(cid); } catch { throw new Error(`statement ${id} is not a valid CIDv1`); } + } + for (const [id, cause] of Object.entries(bindings.causes)) { + validateAddress(cause.owner, `cause ${id} owner`); + if (!cause.refName.trim()) throw new Error(`cause ${id} has an empty ref name`); + try { ensureIpfsCidV1(cause.rosterCid); } catch { throw new Error(`cause ${id} roster is not a valid CIDv1`); } + } + for (const [id, address] of Object.entries(bindings.projects)) validateAddress(address, `project ${id}`); + for (const [id, note] of Object.entries(bindings.notes)) { + validateAddress(note.contractAddress, `note ${id} contract`); + if (!/^(0|[1-9][0-9]*)$/.test(note.noteId)) throw new Error(`note ${id} has an invalid on-chain ID`); + } +} + +function missingBindings(kind: string, plannedIds: Set, values: Record): string[] { + return [...plannedIds].filter((id) => !values[id]).map((id) => `${kind}:${id}`); +} + +function requireCompleteBindings(bindings: CampaignRuntimeBindings, ids: Record<'user' | 'statement' | 'cause' | 'project' | 'note', Set>): void { + const missing = [ + ...missingBindings('user', ids.user, bindings.users), + ...missingBindings('statement', ids.statement, bindings.statements), + ...missingBindings('cause', ids.cause, bindings.causes), + ...missingBindings('project', ids.project, bindings.projects), + ...missingBindings('note', ids.note, bindings.notes), + ]; + if (missing.length > 0) throw new Error(`runtime bindings are incomplete: ${missing.join(', ')}`); +} + +export function validateRuntimeBindings(plan: CampaignPlan, bindings: CampaignRuntimeBindings, options: { complete?: boolean } = {}): void { + if (bindings.version !== CAMPAIGN_RUNTIME_BINDINGS_VERSION || bindings.campaignId !== plan.campaignId || bindings.manifestFingerprint !== plan.manifestFingerprint) { + throw new Error('runtime bindings do not match this campaign plan'); + } + if (!Number.isFinite(Date.parse(bindings.updatedAt))) throw new Error('runtime bindings updatedAt is not a valid timestamp'); + + const userIds = new Set(plan.users.map((item) => item.id)); + const statementIds = new Set(plan.statements.map((item) => item.id)); + const causeIds = new Set(plan.statements.map((item) => item.causeId)); + const projectIds = new Set(plan.projects.map((item) => item.id)); + const noteIds = new Set(plan.actions.flatMap((item) => item.noteId ? [item.noteId] : [])); + validateKeys('users', bindings.users, userIds); + validateKeys('statements', bindings.statements, statementIds); + validateKeys('causes', bindings.causes, causeIds); + validateKeys('projects', bindings.projects, projectIds); + validateKeys('notes', bindings.notes, noteIds); + + validateBindingValues(bindings); + if (options.complete) requireCompleteBindings(bindings, { user: userIds, statement: statementIds, cause: causeIds, project: projectIds, note: noteIds }); +} + +export async function writeRuntimeBindings(plan: CampaignPlan, bindings: CampaignRuntimeBindings, outputPath: string): Promise { + validateRuntimeBindings(plan, bindings); + await mkdir(path.dirname(outputPath), { recursive: true }); + const temporaryPath = `${outputPath}.tmp`; + await writeFile(temporaryPath, `${JSON.stringify(bindings, null, 2)}\n`); + await rename(temporaryPath, outputPath); +} + +export async function loadRuntimeBindings(plan: CampaignPlan, inputPath: string, options: { complete?: boolean } = {}): Promise { + const bindings = JSON.parse(await readFile(inputPath, 'utf8')) as CampaignRuntimeBindings; + validateRuntimeBindings(plan, bindings, options); + return bindings; +} diff --git a/fake-data-generation/test/campaignRuntimeBindings.test.ts b/fake-data-generation/test/campaignRuntimeBindings.test.ts new file mode 100644 index 00000000..0ebc0c21 --- /dev/null +++ b/fake-data-generation/test/campaignRuntimeBindings.test.ts @@ -0,0 +1,55 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { buildCidV1FromDigest } from '@commonality/sdk/utils'; +import { createEmptyRuntimeBindings, loadRuntimeBindings, validateRuntimeBindings, writeRuntimeBindings } from '../campaignRuntimeBindings.js'; +import type { CampaignPlan } from '../campaignPlanner.js'; + +const ADDRESS_A = '0x00000000000000000000000000000000000000a1'; +const ADDRESS_B = '0x00000000000000000000000000000000000000b1'; +const CID = buildCidV1FromDigest(0x70, new Uint8Array(32).fill(7)); +const plan = { + version: 'commonality-campaign-plan-v1', campaignId: 'test', deterministicSeed: 'seed', manifestFingerprint: 'fingerprint', + users: [{ id: 'user-1' }], statements: [{ id: 'statement-1', causeId: 'cause-1' }], projects: [{ id: 'project-1' }], + actions: [{ noteId: 'note-1' }], +} as CampaignPlan; + +function completeBindings() { + const bindings = createEmptyRuntimeBindings(plan, new Date('2026-01-01T00:00:00.000Z')); + bindings.users['user-1'] = ADDRESS_A; + bindings.statements['statement-1'] = CID; + bindings.causes['cause-1'] = { owner: ADDRESS_A, refName: 'cause/test', rosterCid: CID }; + bindings.projects['project-1'] = ADDRESS_B; + bindings.notes['note-1'] = { contractAddress: ADDRESS_B, noteId: '42' }; + return bindings; +} + +test('validates a complete execution-to-SDK binding artifact', () => { + assert.doesNotThrow(() => validateRuntimeBindings(plan, completeBindings(), { complete: true })); +}); + +test('rejects mismatched plans, unknown IDs, invalid values, and incomplete artifacts', () => { + const mismatched = completeBindings(); mismatched.manifestFingerprint = 'other'; + assert.throws(() => validateRuntimeBindings(plan, mismatched), /do not match/); + const unknown = completeBindings(); unknown.users['user-unknown'] = ADDRESS_A; + assert.throws(() => validateRuntimeBindings(plan, unknown), /unknown planned ID/); + const invalid = completeBindings(); invalid.notes['note-1'].noteId = '-1'; + assert.throws(() => validateRuntimeBindings(plan, invalid), /invalid on-chain ID/); + const partial = createEmptyRuntimeBindings(plan); + assert.doesNotThrow(() => validateRuntimeBindings(plan, partial)); + assert.throws(() => validateRuntimeBindings(plan, partial, { complete: true }), /user:user-1.*statement:statement-1.*cause:cause-1.*project:project-1.*note:note-1/); +}); + +test('atomically persists and reloads public bindings without wallet secrets', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-bindings-')); + const outputPath = path.join(directory, 'runtime-bindings.json'); + try { + const bindings = completeBindings(); + await writeRuntimeBindings(plan, bindings, outputPath); + assert.deepEqual(await loadRuntimeBindings(plan, outputPath, { complete: true }), bindings); + const serialized = await readFile(outputPath, 'utf8'); + assert.doesNotMatch(serialized, /privateKey|secret/i); + } finally { await rm(directory, { recursive: true, force: true }); } +}); From 241bb55ada90cf38abe0ed5d5aad629e12566881 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 14:33:59 -0400 Subject: [PATCH 10/19] Add campaign SDK reconciliation checks --- fake-data-generation/README.md | 2 +- .../TESTNET-SIMULATION-PLAN.md | 2 +- .../campaignSdkDerivedChecks.ts | 120 ++++++++++++++++++ .../test/campaignSdkDerivedChecks.test.ts | 79 ++++++++++++ 4 files changed, 201 insertions(+), 2 deletions(-) create mode 100644 fake-data-generation/campaignSdkDerivedChecks.ts create mode 100644 fake-data-generation/test/campaignSdkDerivedChecks.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 7141ec0a..e1d34ec7 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -17,7 +17,7 @@ Campaign execution uses the explicit environment boundary in `campaignEnvironmen The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. No CLI or real contract adapter is exposed yet: this layer is deliberately tested against a local fake chain before the campaign behavior and contract bindings are added. -`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to an explicit SDK-check provider. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. The remaining reconciliation work is to implement the SDK-check provider over that artifact before exercising the deep local stack. +`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. The remaining reconciliation work is to exercise the raw and derived layers against the deep local stack. ## Overview diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 6f016232..1ff6635e 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -182,7 +182,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. -6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter maps every selected action type to its real Ponder event(s), filters by mined transaction, reads chain/indexer heads, and exposes an explicit SDK-derived-check seam. The validated, atomically persisted runtime-binding artifact now maps planned IDs to execution-produced wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs without containing secrets. Next implement the SDK checks over those bindings, then exercise both layers against the deep local stack. +6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter maps every selected action type to its real Ponder event(s), filters by mined transaction, and reads chain/indexer heads. The validated, atomically persisted runtime-binding artifact maps planned IDs to execution-produced wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs without containing secrets. Binding-aware SDK checks now verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and delegation state, including last-write-wins histories. Next exercise both raw and derived layers against the deep local stack. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. diff --git a/fake-data-generation/campaignSdkDerivedChecks.ts b/fake-data-generation/campaignSdkDerivedChecks.ts new file mode 100644 index 00000000..b0561990 --- /dev/null +++ b/fake-data-generation/campaignSdkDerivedChecks.ts @@ -0,0 +1,120 @@ +import type { Address } from 'viem'; +import { BeliefStates, getImplication, getUserBelief } from '@commonality/sdk/conceptspace'; +import { getNote, type Note } from '@commonality/sdk/delegation'; +import { getAlignmentAttestation, PROJECT_ALIGNMENT_TOPIC, toSubjectId } from '@commonality/sdk/fundingportals'; +import { getProject, type Project } from '@commonality/sdk/lazy-giving'; +import type { SDKMachinery } from '@commonality/sdk/machinery'; +import { getRefsByName, type MutableRef } from '@commonality/sdk/mutable-refs'; +import type { IpfsCidV1 } from '@commonality/sdk/utils'; +import type { CampaignDerivedCheckProvider } from './campaignIndexerAdapter.js'; +import type { CampaignPlan, PlannedAction } from './campaignPlanner.js'; +import { validateRuntimeBindings, type CampaignRuntimeBindings } from './campaignRuntimeBindings.js'; +import type { DerivedCheck } from './campaignReconciler.js'; + +/** Injectable SDK query surface, primarily to make the expected-state logic testable. */ +export interface CampaignSdkQueries { + getUserBelief(user: Address, statement: IpfsCidV1): Promise; + hasImplication(attester: Address, from: IpfsCidV1, to: IpfsCidV1): Promise; + getRefsByName(name: string): Promise; + getProject(address: Address): Promise; + hasAlignment(attester: Address, project: Address, statement: IpfsCidV1): Promise; + getNote(noteId: string): Promise; +} + +function realSdkQueries(machinery: SDKMachinery): CampaignSdkQueries { + return { + async getUserBelief(user, statement) { + return (await getUserBelief(machinery, user, statement))?.beliefState ?? BeliefStates.NO_OPINION; + }, + async hasImplication(attester, from, to) { + return (await getImplication(machinery, attester, from, to)) !== null; + }, + getRefsByName: (name) => getRefsByName(machinery, name, 10_000), + getProject: (address) => getProject(machinery, address), + async hasAlignment(attester, project, statement) { + return (await getAlignmentAttestation(machinery, attester, toSubjectId(project), statement, PROJECT_ALIGNMENT_TOPIC)) !== null; + }, + getNote: (noteId) => getNote(machinery, noteId), + }; +} + +function check(name: string, expected: DerivedCheck['expected'], actual: DerivedCheck['actual']): DerivedCheck { + return { name, expected, actual }; +} + +/** + * Build SDK-fold checks against the campaign's final intended state. Historical + * writes to the same belief, project, or note deliberately share the final + * expectation because reconciliation runs after the complete campaign settles. + */ +export function createCampaignSdkDerivedCheckProvider(input: { + machinery: SDKMachinery; + plan: CampaignPlan; + bindings: CampaignRuntimeBindings; + queries?: CampaignSdkQueries; +}): CampaignDerivedCheckProvider { + const { plan, bindings } = input; + validateRuntimeBindings(plan, bindings, { complete: true }); + const queries = input.queries ?? realSdkQueries(input.machinery); + const latestBelief = new Map(); + const latestNoteAction = new Map(); + const fundingByProject = new Map(); + for (const action of plan.actions) { + if (action.type === 'set-belief') latestBelief.set(`${action.actorUserId}/${action.statementId}`, action); + if (action.noteId) latestNoteAction.set(action.noteId, action); + if (action.type === 'fund-project') fundingByProject.set(action.projectId!, (fundingByProject.get(action.projectId!) ?? 0) + action.amount!); + } + + const user = (id: string | null | undefined): Address => bindings.users[id!]; + const statement = (id: string | undefined): IpfsCidV1 => bindings.statements[id!]; + const project = (id: string | undefined): Address => bindings.projects[id!]; + + return { + // One explicit branch per frozen campaign action type keeps coverage auditable. + // eslint-disable-next-line complexity + async getDerivedChecks(action): Promise { + switch (action.type) { + case 'publish-statement': + // DataPublished has no independent SDK fold; its raw indexed event is + // the authoritative publication check. Later actions query this CID. + return []; + case 'create-cause': { + const binding = bindings.causes[action.causeId!]; + const refs = await queries.getRefsByName(binding.refName); + const actual = refs.find((ref) => ref.owner.toLowerCase() === binding.owner.toLowerCase())?.value ?? null; + return [check('SDK cause roster ref', binding.rosterCid, actual)]; + } + case 'set-belief': { + const final = latestBelief.get(`${action.actorUserId}/${action.statementId}`)!; + const expected = final.belief === 'believe' ? BeliefStates.BELIEVES : BeliefStates.DISBELIEVES; + return [check('SDK final user belief', expected, await queries.getUserBelief(user(action.actorUserId), statement(action.statementId)))]; + } + case 'attest-implication': + return [check('SDK active implication', true, await queries.hasImplication(user(action.actorUserId), statement(action.implication!.fromStatementId), statement(action.implication!.toStatementId)))]; + case 'create-project': + return [check('SDK project exists', true, (await queries.getProject(project(action.projectId))) !== null)]; + case 'attest-alignment': + return [check('SDK active project alignment', true, await queries.hasAlignment(user(action.actorUserId), project(action.projectId), statement(action.statementId)))]; + case 'fund-project': { + const folded = await queries.getProject(project(action.projectId)); + return [check('SDK final project funding', String(fundingByProject.get(action.projectId!) ?? 0), folded?.totalReceived ?? null)]; + } + case 'deposit-note': + case 'delegate-note': + case 'revoke-delegation': { + const binding = bindings.notes[action.noteId!]; + const folded = await queries.getNote(`${binding.contractAddress}:${binding.noteId}`); + const final = latestNoteAction.get(action.noteId!)!; + const expectedOwner = final.type === 'delegate-note' ? user(final.delegateUserId) : user(final.actorUserId); + const rootOwner = user(plan.actions.find((item) => item.type === 'deposit-note' && item.noteId === action.noteId)!.actorUserId); + return [ + check('SDK note exists', true, folded !== null), + check('SDK final note owner', expectedOwner.toLowerCase(), folded?.owner.toLowerCase() ?? null), + check('SDK note root owner', rootOwner.toLowerCase(), folded?.rootOwner.toLowerCase() ?? null), + check('SDK note remains active', true, folded?.active ?? null), + ]; + } + } + }, + }; +} diff --git a/fake-data-generation/test/campaignSdkDerivedChecks.test.ts b/fake-data-generation/test/campaignSdkDerivedChecks.test.ts new file mode 100644 index 00000000..4d06f658 --- /dev/null +++ b/fake-data-generation/test/campaignSdkDerivedChecks.test.ts @@ -0,0 +1,79 @@ +import assert from 'node:assert/strict'; +import { readFile } from 'node:fs/promises'; +import test from 'node:test'; +import type { Address } from 'viem'; +import { fakeIpfsCidV1 } from '@commonality/sdk/testing'; +import { createCampaignSdkDerivedCheckProvider, type CampaignSdkQueries } from '../campaignSdkDerivedChecks.js'; +import { buildCampaignPlan } from '../campaignPlanner.js'; +import { CAMPAIGN_RUNTIME_BINDINGS_VERSION, type CampaignRuntimeBindings } from '../campaignRuntimeBindings.js'; +import type { CampaignManifestV1 } from '../campaignSchema.js'; + +const address = (index: number): Address => `0x${index.toString(16).padStart(40, '0')}` as Address; + +test('SDK provider checks final folded state through runtime bindings', async () => { + const manifest = JSON.parse(await readFile(new URL('../campaigns/medium-realistic-v1.json', import.meta.url), 'utf8')) as CampaignManifestV1; + const plan = await buildCampaignPlan(manifest); + const bindings: CampaignRuntimeBindings = { + version: CAMPAIGN_RUNTIME_BINDINGS_VERSION, + campaignId: plan.campaignId, + manifestFingerprint: plan.manifestFingerprint, + updatedAt: new Date(0).toISOString(), + users: Object.fromEntries(plan.users.map((item, index) => [item.id, address(index + 1)])), + statements: Object.fromEntries(plan.statements.map((item) => [item.id, fakeIpfsCidV1(item.id)])), + causes: Object.fromEntries([...new Set(plan.statements.map((item) => item.causeId))].map((id, index) => [id, { owner: address(index + 201), refName: `cause-${id}`, rosterCid: fakeIpfsCidV1(`roster-${id}`) }])), + projects: Object.fromEntries(plan.projects.map((item, index) => [item.id, address(index + 301)])), + notes: Object.fromEntries([...new Set(plan.actions.flatMap((item) => item.noteId ? [item.noteId] : []))].map((id, index) => [id, { contractAddress: address(401), noteId: String(index + 1) }])), + }; + + const calls: string[] = []; + const latestBeliefs = new Map(); + for (const action of plan.actions.filter((item) => item.type === 'set-belief')) { + latestBeliefs.set(`${bindings.users[action.actorUserId!]}/${bindings.statements[action.statementId!]}`, action.belief === 'believe' ? 1 : 2); + } + const queries: CampaignSdkQueries = { + getUserBelief: async (user, statement) => latestBeliefs.get(`${user}/${statement}`)!, + hasImplication: async () => true, + getRefsByName: async (name) => { + const binding = Object.values(bindings.causes).find((item) => item.refName === name)!; + return [{ owner: binding.owner, name, value: binding.rosterCid, updatedAt: '0', updatedAtBlock: '1', transactionHash: `0x${'1'.repeat(64)}` }]; + }, + getProject: async (projectAddress) => { + calls.push(projectAddress); + const projectId = Object.entries(bindings.projects).find(([, value]) => value === projectAddress)![0]; + const total = plan.actions.filter((item) => item.type === 'fund-project' && item.projectId === projectId).reduce((sum, item) => sum + item.amount!, 0); + return { id: projectAddress, erc1155Address: address(999), marketplaceAddress: null, recipient: address(998), fundingCurrency: { chainId: 31337, tokenAddress: address(997), symbol: 'TEST', decimals: 0 }, threshold: '1', deadline: '1', totalReceived: String(total), conditionAddress: null }; + }, + hasAlignment: async () => true, + getNote: async (scopedId) => { + const bindingEntry = Object.entries(bindings.notes).find(([, value]) => `${value.contractAddress}:${value.noteId}` === scopedId)!; + const deposit = plan.actions.find((item) => item.type === 'deposit-note' && item.noteId === bindingEntry[0])!; + const final = plan.actions.filter((item) => item.noteId === bindingEntry[0]).at(-1)!; + const owner = final.type === 'delegate-note' ? bindings.users[final.delegateUserId!] : bindings.users[final.actorUserId!]; + return { id: bindingEntry[1].noteId, contractAddress: bindingEntry[1].contractAddress, chainHash: '0x', amount: String(deposit.amount), token: address(996), tokenType: 0, tokenId: '0', owner, rootOwner: bindings.users[deposit.actorUserId!], active: true, createdAt: '0', createdAtBlock: '1', updatedAt: '0' }; + }, + }; + const provider = createCampaignSdkDerivedCheckProvider({ machinery: { ipfsConfig: {}, twitterApiConfig: {}, testConfig: {} }, plan, bindings, queries }); + + for (const action of plan.actions) { + const checks = await provider.getDerivedChecks(action); + assert.ok(checks.every((item) => item.expected === item.actual), `${action.id} ${action.type} did not match`); + } + assert.ok(calls.length > 0); +}); + +test('SDK provider exposes a derived mismatch instead of hiding it', async () => { + const manifest = JSON.parse(await readFile(new URL('../campaigns/medium-realistic-v1.json', import.meta.url), 'utf8')) as CampaignManifestV1; + const plan = await buildCampaignPlan(manifest); + const users = Object.fromEntries(plan.users.map((item, index) => [item.id, address(index + 1)])); + const bindings = { + version: CAMPAIGN_RUNTIME_BINDINGS_VERSION, campaignId: plan.campaignId, manifestFingerprint: plan.manifestFingerprint, updatedAt: new Date(0).toISOString(), users, + statements: Object.fromEntries(plan.statements.map((item) => [item.id, fakeIpfsCidV1(item.id)])), + causes: Object.fromEntries([...new Set(plan.statements.map((item) => item.causeId))].map((id, index) => [id, { owner: address(index + 201), refName: id, rosterCid: fakeIpfsCidV1(id) }])), + projects: Object.fromEntries(plan.projects.map((item, index) => [item.id, address(index + 301)])), + notes: Object.fromEntries([...new Set(plan.actions.flatMap((item) => item.noteId ? [item.noteId] : []))].map((id, index) => [id, { contractAddress: address(401), noteId: String(index + 1) }])), + } satisfies CampaignRuntimeBindings; + const queries = { getUserBelief: async () => 0 } as CampaignSdkQueries; + const provider = createCampaignSdkDerivedCheckProvider({ machinery: { ipfsConfig: {}, twitterApiConfig: {}, testConfig: {} }, plan, bindings, queries }); + const belief = plan.actions.find((item) => item.type === 'set-belief')!; + assert.notEqual((await provider.getDerivedChecks(belief))[0].expected, (await provider.getDerivedChecks(belief))[0].actual); +}); From c7f2d6a0e6c13832d6510d39b3a6fc332c42f79e Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 14:41:48 -0400 Subject: [PATCH 11/19] Exercise campaign reconciliation against the local stack Paginate raw indexer lookups, collapse alternative proving events, and add gen:campaign:reconcile plus a live adapter probe. --- CONTINUITY.md | 4 + fake-data-generation/README.md | 2 +- .../TESTNET-SIMULATION-PLAN.md | 2 +- .../campaignIndexerAdapter.ts | 23 +++++- fake-data-generation/campaignLocalStack.ts | 77 +++++++++++++++++++ fake-data-generation/campaignPlanner.ts | 24 +++++- fake-data-generation/campaignReconcile.ts | 29 +++++++ fake-data-generation/package.json | 1 + .../test/campaignIndexerAdapter.test.ts | 16 +++- .../test/campaignLocalStack.live.test.ts | 59 ++++++++++++++ .../test/campaignPlanner.test.ts | 17 +++- 11 files changed, 244 insertions(+), 10 deletions(-) create mode 100644 fake-data-generation/campaignLocalStack.ts create mode 100644 fake-data-generation/campaignReconcile.ts create mode 100644 fake-data-generation/test/campaignLocalStack.live.test.ts diff --git a/CONTINUITY.md b/CONTINUITY.md index 49a9ec28..0635636e 100644 --- a/CONTINUITY.md +++ b/CONTINUITY.md @@ -2147,3 +2147,7 @@ not more protocol surface. Removed the item from `focus.md`. Spec status is closed. Deferred: indexed beneficiary lookup and bulk/bot UX. + +## 2026-09-11 — Campaign reconciliation talks to the local stack + +Item 6 of the medium-realistic campaign: indexer adapter paginates with `fetchEventsComplete` and collapses alternative proving events so ordinary vs retroactive funding is one write. `npm run gen:campaign:reconcile -- --probe` reads live chain/indexer heads. Without `--probe` it reconciles from plan + execution + runtime-bindings. A live adapter test exercises raw lookup plus an SDK belief fold against Ponder. Campaign execution adapter is still required before mined campaign txs exist to close item 6. diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index e1d34ec7..6a239e53 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -17,7 +17,7 @@ Campaign execution uses the explicit environment boundary in `campaignEnvironmen The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. No CLI or real contract adapter is exposed yet: this layer is deliberately tested against a local fake chain before the campaign behavior and contract bindings are added. -`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. The remaining reconciliation work is to exercise the raw and derived layers against the deep local stack. +`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. `npm run gen:campaign:reconcile -- --probe` talks to the live local chain and Ponder `/status` heads. The same command without `--probe` loads planned actions, execution state, and runtime bindings, then writes the reconciliation JSON/summary through the real indexer adapter and SDK folds. Raw lookups paginate with `fetchEventsComplete` and treat ordinary vs retroactive funding events as one proving write. A live adapter test runs when `http://localhost:42069` is reachable. Full verification of mined campaign actions still waits on the local campaign execution adapter. ## Overview diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 1ff6635e..c9284421 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -182,7 +182,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. -6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter maps every selected action type to its real Ponder event(s), filters by mined transaction, and reads chain/indexer heads. The validated, atomically persisted runtime-binding artifact maps planned IDs to execution-produced wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs without containing secrets. Binding-aware SDK checks now verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and delegation state, including last-write-wins histories. Next exercise both raw and derived layers against the deep local stack. +6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter maps every selected action type to its real Ponder event(s), paginates through `fetchEventsComplete`, treats alternative proving events as one write, filters by mined transaction, and reads chain/indexer heads. The validated, atomically persisted runtime-binding artifact maps planned IDs to execution-produced wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs without containing secrets. Binding-aware SDK checks now verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and delegation state, including last-write-wins histories. `npm run gen:campaign:reconcile` probes the live local stack and can reconcile from artifacts; a live adapter test exercises raw lookup plus an SDK belief fold against Ponder when the indexer is up. Next: bind execution so mined campaign transactions exist, then reconcile a real local run. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. diff --git a/fake-data-generation/campaignIndexerAdapter.ts b/fake-data-generation/campaignIndexerAdapter.ts index 90d7ac56..c6b86e7d 100644 --- a/fake-data-generation/campaignIndexerAdapter.ts +++ b/fake-data-generation/campaignIndexerAdapter.ts @@ -1,6 +1,6 @@ import type { PublicClient } from 'viem'; import type { SDKMachinery } from '@commonality/sdk/machinery'; -import { chainStatusKeyForChainId, fetchEvents, type RawEventFromCache } from '@commonality/sdk/utils'; +import { chainStatusKeyForChainId, fetchEventsComplete, type RawEventFromCache } from '@commonality/sdk/utils'; import type { CampaignActionType } from './campaignSchema.js'; import type { PlannedAction } from './campaignPlanner.js'; import type { CampaignReconciliationAdapter, DerivedCheck, IndexedActionMatch } from './campaignReconciler.js'; @@ -52,6 +52,23 @@ function toMatch(event: RawEventFromCache): IndexedActionMatch { }; } +/** + * Alternative proving events (e.g. ordinary vs retroactive funding) count as + * one indexed write. Duplicates of the same event name remain visible. + */ +export function collapseIndexedMatches(events: readonly RawEventFromCache[]): IndexedActionMatch[] { + const byName = new Map(); + for (const event of events) { + const group = byName.get(event.eventName) ?? []; + group.push(event); + byName.set(event.eventName, group); + } + for (const group of byName.values()) { + if (group.length > 1) return group.map(toMatch); + } + return events[0] ? [toMatch(events[0])] : []; +} + /** * Bind campaign reconciliation to the real Ponder event cache and SDK fold seam. * Event-cache results are filtered by transaction hash because the public API @@ -68,9 +85,9 @@ export function createCampaignIndexerAdapter(input: { getIndexerHead: () => getIndexerHead(input.machinery), async findIndexedAction(action, transactionHash) { const eventGroups = await Promise.all(CAMPAIGN_ACTION_EVENTS[action.type].map((eventName) => - fetchEvents(input.machinery, { eventName, limit: 1000 }))); + fetchEventsComplete(input.machinery, { eventName }))); const target = transactionHash.toLowerCase(); - return eventGroups.flat().filter((event) => event.transactionHash.toLowerCase() === target).map(toMatch); + return collapseIndexedMatches(eventGroups.flat().filter((event) => event.transactionHash.toLowerCase() === target)); }, getDerivedChecks: (action) => input.derivedChecks.getDerivedChecks(action), }; diff --git a/fake-data-generation/campaignLocalStack.ts b/fake-data-generation/campaignLocalStack.ts new file mode 100644 index 00000000..bf66a67f --- /dev/null +++ b/fake-data-generation/campaignLocalStack.ts @@ -0,0 +1,77 @@ +import { readFile } from 'node:fs/promises'; +import path from 'node:path'; +import { createPublicClient, http } from 'viem'; +import { createSDKMachinery, type SDKMachinery } from '@commonality/sdk/machinery'; +import { CAMPAIGN_EXECUTION_VERSION, type CampaignExecutionState } from './campaignExecutor.js'; +import { createCampaignIndexerAdapter } from './campaignIndexerAdapter.js'; +import { loadCampaignPlan } from './campaignPlanner.js'; +import { formatReconciliationReport, reconcileCampaign, writeReconciliationArtifacts, type CampaignReconciliationReport } from './campaignReconciler.js'; +import { loadRuntimeBindings } from './campaignRuntimeBindings.js'; +import { type CampaignManifestV1 } from './campaignSchema.js'; +import { createCampaignSdkDerivedCheckProvider } from './campaignSdkDerivedChecks.js'; +import { RPC_URL } from './loadEnv.js'; + +export const DEFAULT_EVENT_CACHE_URL = 'http://localhost:42069'; + +export interface LocalCampaignStack { + machinery: SDKMachinery; + publicClient: ReturnType; +} + +export function createLocalCampaignStack(input: { rpcUrl?: string; eventCacheUrl?: string; chainId?: number } = {}): LocalCampaignStack { + const rpcUrl = input.rpcUrl ?? RPC_URL; + const eventCacheUrl = input.eventCacheUrl ?? process.env.EVENT_CACHE_URL ?? DEFAULT_EVENT_CACHE_URL; + const chainId = input.chainId ?? 31_337; + const publicClient = createPublicClient({ transport: http(rpcUrl) }); + return { + publicClient, + machinery: createSDKMachinery({ + eventCacheUrl, + defaultChainId: chainId, + chainStatusKey: chainId === 31_337 ? 'hardhat' : undefined, + publicClient, + }), + }; +} + +export async function probeLocalCampaignStack(stack: LocalCampaignStack): Promise<{ chainHead: bigint; indexerHead: bigint }> { + const adapter = createCampaignIndexerAdapter({ + machinery: stack.machinery, + publicClient: stack.publicClient, + derivedChecks: { getDerivedChecks: async () => [] }, + }); + const [chainHead, indexerHead] = await Promise.all([adapter.getChainHead(), adapter.getIndexerHead()]); + return { chainHead, indexerHead }; +} + +export async function reconcileLocalCampaign(input: { + manifest: CampaignManifestV1; + outputDirectory: string; + bindingsPath: string; + executionPath?: string; + stack?: LocalCampaignStack; + settlingWindowMs?: number; + pollIntervalMs?: number; +}): Promise<{ report: CampaignReconciliationReport; summary: string; jsonPath: string; summaryPath: string }> { + const stack = input.stack ?? createLocalCampaignStack(); + const plan = await loadCampaignPlan(input.manifest, input.outputDirectory); + const bindings = await loadRuntimeBindings(plan, input.bindingsPath, { complete: true }); + const executionPath = input.executionPath ?? path.join(input.outputDirectory, input.manifest.artifactLayout.executionState); + const execution = JSON.parse(await readFile(executionPath, 'utf8')) as CampaignExecutionState; + if (execution.version !== CAMPAIGN_EXECUTION_VERSION) throw new Error('execution state version is not campaign-execution-v1'); + const adapter = createCampaignIndexerAdapter({ + machinery: stack.machinery, + publicClient: stack.publicClient, + derivedChecks: createCampaignSdkDerivedCheckProvider({ machinery: stack.machinery, plan, bindings }), + }); + const report = await reconcileCampaign({ + actions: plan.actions, + execution, + adapter, + options: { settlingWindowMs: input.settlingWindowMs ?? 30_000, pollIntervalMs: input.pollIntervalMs ?? 1_000 }, + }); + const jsonPath = path.join(input.outputDirectory, input.manifest.artifactLayout.reconciliation); + const summaryPath = path.join(input.outputDirectory, input.manifest.artifactLayout.summary); + await writeReconciliationArtifacts(report, jsonPath, summaryPath); + return { report, summary: formatReconciliationReport(report), jsonPath, summaryPath }; +} diff --git a/fake-data-generation/campaignPlanner.ts b/fake-data-generation/campaignPlanner.ts index 8cf21732..9830a80d 100644 --- a/fake-data-generation/campaignPlanner.ts +++ b/fake-data-generation/campaignPlanner.ts @@ -286,7 +286,29 @@ export function validatePlannedActions(manifest: CampaignManifestV1, statements: if (new Set(users.map((user) => user.walletSlot)).size !== users.length) throw new Error('planned wallet slots must be unique'); } -async function writePlanArtifacts(manifest: CampaignManifestV1, plan: CampaignPlan, outputDirectory: string): Promise { +export async function loadCampaignPlan(manifest: CampaignManifestV1, outputDirectory: string): Promise { + const readJson = async (relativePath: string): Promise> => + JSON.parse(await readFile(path.join(outputDirectory, relativePath), 'utf8')) as Record; + const statementCatalog = await readJson(manifest.artifactLayout.statementCatalog); + const assignments = await readJson(manifest.artifactLayout.assignments); + const actionPlan = await readJson(manifest.artifactLayout.actionPlan); + const plan: CampaignPlan = { + version: CAMPAIGN_PLAN_VERSION, + campaignId: manifest.campaign.id, + deterministicSeed: manifest.campaign.deterministicSeed, + manifestFingerprint: String(actionPlan.manifestFingerprint), + statements: statementCatalog.statements as CampaignPlan['statements'], + users: assignments.users as CampaignPlan['users'], + projects: assignments.projects as CampaignPlan['projects'], + actions: actionPlan.actions as CampaignPlan['actions'], + estimate: actionPlan.estimate as CampaignPlan['estimate'], + }; + if (plan.campaignId !== actionPlan.campaignId) throw new Error('action-plan campaign ID does not match the manifest'); + validatePlannedActions(manifest, plan.statements, plan.users, plan.projects, plan.actions); + return plan; +} + +export async function writePlanArtifacts(manifest: CampaignManifestV1, plan: CampaignPlan, outputDirectory: string): Promise { const writeJson = async (relativePath: string, value: JsonValue | object): Promise => { const target = path.join(outputDirectory, relativePath); await mkdir(path.dirname(target), { recursive: true }); await writeFile(target, `${JSON.stringify(value, null, 2)}\n`); }; await writeJson(manifest.artifactLayout.manifest, manifest); await writeJson(manifest.artifactLayout.statementCatalog, { version: plan.version, campaignId: plan.campaignId, statements: plan.statements }); diff --git a/fake-data-generation/campaignReconcile.ts b/fake-data-generation/campaignReconcile.ts new file mode 100644 index 00000000..fe5b38b6 --- /dev/null +++ b/fake-data-generation/campaignReconcile.ts @@ -0,0 +1,29 @@ +import { readFile } from 'node:fs/promises'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { createLocalCampaignStack, probeLocalCampaignStack, reconcileLocalCampaign } from './campaignLocalStack.js'; +import type { CampaignManifestV1 } from './campaignSchema.js'; +import { loadEnv } from './loadEnv.js'; + +loadEnv(); + +async function main(): Promise { + const directory = path.dirname(fileURLToPath(import.meta.url)); + const probeOnly = process.argv.includes('--probe'); + const args = process.argv.slice(2).filter((arg) => arg !== '--probe'); + const manifestPath = args[0] ?? path.join(directory, 'campaigns/medium-realistic-v1.json'); + const outputDirectory = args[1] ?? path.join(directory, 'output/campaigns/medium-realistic-v1'); + const bindingsPath = args[2] ?? path.join(outputDirectory, 'execution/runtime-bindings.json'); + const stack = createLocalCampaignStack(); + const heads = await probeLocalCampaignStack(stack); + console.log(`Chain head ${heads.chainHead}; indexer head ${heads.indexerHead}; lag ${heads.chainHead > heads.indexerHead ? heads.chainHead - heads.indexerHead : 0n} blocks.`); + if (probeOnly) return; + const manifest = JSON.parse(await readFile(manifestPath, 'utf8')) as CampaignManifestV1; + const result = await reconcileLocalCampaign({ manifest, outputDirectory, bindingsPath, stack }); + console.log(result.summary); + console.log(`Wrote ${result.jsonPath} and ${result.summaryPath}`); +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main().catch((error) => { console.error(error instanceof Error ? error.message : error); process.exitCode = 1; }); +} diff --git a/fake-data-generation/package.json b/fake-data-generation/package.json index c256586a..82b7d968 100644 --- a/fake-data-generation/package.json +++ b/fake-data-generation/package.json @@ -18,6 +18,7 @@ "test:seed:implication-regression": "tsx verifySeedImplicationEvaluations.ts", "gen:proliferation": "tsx generateProliferation.ts", "gen:campaign:plan": "tsx campaignPlanner.ts", + "gen:campaign:reconcile": "tsx campaignReconcile.ts", "gen:attesters": "tsx generateAttesters.ts", "gen:attestations": "tsx generateAttestations.ts", "gen:simulate": "tsx runSimulation.ts", diff --git a/fake-data-generation/test/campaignIndexerAdapter.test.ts b/fake-data-generation/test/campaignIndexerAdapter.test.ts index 81ef3474..814282df 100644 --- a/fake-data-generation/test/campaignIndexerAdapter.test.ts +++ b/fake-data-generation/test/campaignIndexerAdapter.test.ts @@ -1,6 +1,6 @@ import assert from 'node:assert/strict'; import test from 'node:test'; -import { createCampaignIndexerAdapter, CAMPAIGN_ACTION_EVENTS } from '../campaignIndexerAdapter.js'; +import { createCampaignIndexerAdapter, CAMPAIGN_ACTION_EVENTS, collapseIndexedMatches } from '../campaignIndexerAdapter.js'; import type { PlannedAction } from '../campaignPlanner.js'; const TX = `0x${'a'.repeat(64)}` as const; @@ -36,14 +36,24 @@ test('adapter reads chain/indexer heads and matches only the action transaction' assert.equal(await adapter.getChainHead(), 43n); assert.equal(await adapter.getIndexerHead(), 41n); const matches = await adapter.findIndexedAction(action, TX); - assert.equal(matches.length, 2); - assert.deepEqual(matches.map((match) => match.eventId).sort(), ['ERC1155Bought:ERC1155Bought-wanted', 'RetroactiveDonationReceived:RetroactiveDonationReceived-wanted']); + assert.equal(matches.length, 1); + assert.ok(matches[0].eventId.endsWith('-wanted')); assert.deepEqual(await adapter.getDerivedChecks(action), [{ name: 'SDK project funding', expected: 10, actual: 10 }]); assert.ok(requested.some((url) => url.includes('eventName=ERC1155Bought'))); assert.ok(requested.some((url) => url.includes('eventName=RetroactiveDonationReceived'))); + assert.ok(requested.some((url) => url.includes('limit=10000'))); } finally { globalThis.fetch = originalFetch; } }); +test('alternative proving events collapse; same-name logs stay duplicates', () => { + const event = (eventName: string, id: string) => ({ + id, contractAddress: '0x1', eventName, blockNumber: '1', blockTimestamp: '1', transactionHash: TX, logIndex: 0, + topic0: null, topic1: null, topic2: null, topic3: null, data: '0x', + }); + assert.equal(collapseIndexedMatches([event('ERC1155Bought', 'a'), event('RetroactiveDonationReceived', 'b')]).length, 1); + assert.equal(collapseIndexedMatches([event('ERC1155Bought', 'a'), event('ERC1155Bought', 'b')]).length, 2); +}); + test('adapter fails closed when indexer status omits the configured chain', async () => { const originalFetch = globalThis.fetch; globalThis.fetch = (async () => new Response(JSON.stringify({}))) as typeof fetch; diff --git a/fake-data-generation/test/campaignLocalStack.live.test.ts b/fake-data-generation/test/campaignLocalStack.live.test.ts new file mode 100644 index 00000000..731a58a7 --- /dev/null +++ b/fake-data-generation/test/campaignLocalStack.live.test.ts @@ -0,0 +1,59 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { buildCidV1FromDigest } from '@commonality/sdk/utils'; +import { createCampaignIndexerAdapter } from '../campaignIndexerAdapter.js'; +import { createLocalCampaignStack, probeLocalCampaignStack } from '../campaignLocalStack.js'; +import type { PlannedAction } from '../campaignPlanner.js'; +import { createCampaignSdkDerivedCheckProvider } from '../campaignSdkDerivedChecks.js'; +import { createEmptyRuntimeBindings } from '../campaignRuntimeBindings.js'; +import type { CampaignPlan } from '../campaignPlanner.js'; + +const TX = `0x${'c'.repeat(64)}` as const; +const ADDRESS = '0x00000000000000000000000000000000000000a1' as const; +const CID = buildCidV1FromDigest(0x70, new Uint8Array(32).fill(3)); + +async function liveStackAvailable(): Promise { + try { + const response = await fetch('http://localhost:42069/status', { signal: AbortSignal.timeout(1500) }); + return response.ok; + } catch { + return false; + } +} + +test('raw indexer adapter and SDK derived checks talk to the live local stack', async (t) => { + if (!await liveStackAvailable()) { + t.skip('local indexer is not reachable at http://localhost:42069'); + return; + } + const stack = createLocalCampaignStack(); + const heads = await probeLocalCampaignStack(stack); + assert.ok(heads.chainHead >= 0n); + assert.ok(heads.indexerHead >= 0n); + + const action: PlannedAction = { + id: 'action-00001', sequence: 1, type: 'set-belief', actorUserId: 'user-1', statementId: 'statement-1', + belief: 'believe', dependsOn: [], + }; + const plan = { + version: 'commonality-campaign-plan-v1', campaignId: 'live-probe', deterministicSeed: 'seed', manifestFingerprint: 'fp', + statements: [{ id: 'statement-1', causeId: 'cause-1' }], users: [{ id: 'user-1' }], projects: [], + actions: [action], + } as CampaignPlan; + const bindings = createEmptyRuntimeBindings(plan, new Date('2026-01-01T00:00:00.000Z')); + bindings.users['user-1'] = ADDRESS; + bindings.statements['statement-1'] = CID; + bindings.causes['cause-1'] = { owner: ADDRESS, refName: 'cause/live-probe', rosterCid: CID }; + + const adapter = createCampaignIndexerAdapter({ + machinery: stack.machinery, + publicClient: stack.publicClient, + derivedChecks: createCampaignSdkDerivedCheckProvider({ machinery: stack.machinery, plan, bindings }), + }); + assert.deepEqual(await adapter.findIndexedAction(action, TX), []); + const derived = await adapter.getDerivedChecks(action); + assert.equal(derived.length, 1); + assert.equal(derived[0].name, 'SDK final user belief'); + assert.equal(derived[0].expected, 1); + assert.equal(derived[0].actual, 0); +}); diff --git a/fake-data-generation/test/campaignPlanner.test.ts b/fake-data-generation/test/campaignPlanner.test.ts index 78aef30f..21b4b950 100644 --- a/fake-data-generation/test/campaignPlanner.test.ts +++ b/fake-data-generation/test/campaignPlanner.test.ts @@ -2,7 +2,10 @@ import assert from 'node:assert/strict'; import { readFile } from 'node:fs/promises'; import test from 'node:test'; import type { CampaignManifestV1 } from '../campaignSchema.js'; -import { buildCampaignPlan, validatePlannedActions } from '../campaignPlanner.js'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { buildCampaignPlan, loadCampaignPlan, validatePlannedActions, writePlanArtifacts } from '../campaignPlanner.js'; async function loadManifest(): Promise { return JSON.parse(await readFile(new URL('../campaigns/medium-realistic-v1.json', import.meta.url), 'utf8')) as CampaignManifestV1; @@ -86,3 +89,15 @@ test('validation rejects an action whose prerequisite points forward', async () invalid[0].dependsOn = [invalid[1].id]; assert.throws(() => validatePlannedActions(manifest, plan.statements, plan.users, plan.projects, invalid), /impossible dependency/); }); + +test('written planning artifacts reload into the same campaign plan', async () => { + const manifest = await loadManifest(); + const plan = await buildCampaignPlan(manifest); + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-plan-')); + try { + await writePlanArtifacts(manifest, plan, directory); + assert.deepEqual(await loadCampaignPlan(manifest, directory), plan); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); From c2c0b2797856373d8287c2ad7256bbcfd6fdc357 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 14:52:30 -0400 Subject: [PATCH 12/19] Bind campaign execution to contract writes. Add a local-first adapter and CLI that submit planned campaign actions, persist mined hashes, and update public runtime bindings without storing secrets. --- fake-data-generation/README.md | 4 +- .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignActionAdapter.ts | 240 ++++++++++++++++++ fake-data-generation/campaignExecute.ts | 126 +++++++++ fake-data-generation/campaignSchema.ts | 2 +- .../campaigns/medium-realistic-v1.json | 1 + fake-data-generation/package.json | 1 + .../test/campaignActionAdapter.test.ts | 110 ++++++++ 8 files changed, 482 insertions(+), 4 deletions(-) create mode 100644 fake-data-generation/campaignActionAdapter.ts create mode 100644 fake-data-generation/campaignExecute.ts create mode 100644 fake-data-generation/test/campaignActionAdapter.test.ts diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 6a239e53..1c3c33ad 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -15,9 +15,9 @@ The medium realistic campaign has a separate planning-only command: `npm run gen Campaign execution uses the explicit environment boundary in `campaignEnvironment.ts`. Local mode is pinned to Hardhat chain 31337 and may use the legacy deploy/Hardhat-wallet/free-mint conveniences. Remote mode requires a committed deployment env manifest, an explicit non-Hardhat chain ID, existing bytecode at every required address, generated campaign wallets, and transfer-only payment-token provisioning. Deployment and provisioning adapters are validated against those policies before they can run; remote execution must also carry a separate mutation-confirmation flag when it is added. The existing `gen:tiny`/`gen:small`/`gen:medium`/`gen:large` commands remain local-only legacy simulation paths and are not remote campaign commands. -The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. No CLI or real contract adapter is exposed yet: this layer is deliberately tested against a local fake chain before the campaign behavior and contract bindings are added. +The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) is the contract-binding layer: it maps each planned action type onto SDK writes, classifies RPC vs revert failures, and updates public runtime bindings (no private keys). `npm run gen:campaign:execute` runs that adapter locally after environment preflight; remote mode still requires `--confirm-remote-mutation`. Generated wallet secrets stay under `output/campaigns/secrets/` (gitignored via `output/`). -`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. `npm run gen:campaign:reconcile -- --probe` talks to the live local chain and Ponder `/status` heads. The same command without `--probe` loads planned actions, execution state, and runtime bindings, then writes the reconciliation JSON/summary through the real indexer adapter and SDK folds. Raw lookups paginate with `fetchEventsComplete` and treat ordinary vs retroactive funding events as one proving write. A live adapter test runs when `http://localhost:42069` is reachable. Full verification of mined campaign actions still waits on the local campaign execution adapter. +`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. `npm run gen:campaign:reconcile -- --probe` talks to the live local chain and Ponder `/status` heads. The same command without `--probe` loads planned actions, execution state, and runtime bindings, then writes the reconciliation JSON/summary through the real indexer adapter and SDK folds. Raw lookups paginate with `fetchEventsComplete` and treat ordinary vs retroactive funding events as one proving write. A live adapter test runs when `http://localhost:42069` is reachable. Full 100-user local execution still needs provisioning for wallets beyond the funded Hardhat accounts; a dry adapter test covers binding updates without a live chain. ## Overview diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index c9284421..07ae83ec 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -182,7 +182,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. -6. **[ ] Build transaction-to-indexer deep reconciliation.** The reusable reconciliation engine and adapter contract cover per-action raw matches, derived checks, duplicates, chain-head lag, indexing latency, bounded settling, machine-readable results, and a concise human report. The concrete indexer adapter maps every selected action type to its real Ponder event(s), paginates through `fetchEventsComplete`, treats alternative proving events as one write, filters by mined transaction, and reads chain/indexer heads. The validated, atomically persisted runtime-binding artifact maps planned IDs to execution-produced wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs without containing secrets. Binding-aware SDK checks now verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and delegation state, including last-write-wins histories. `npm run gen:campaign:reconcile` probes the live local stack and can reconcile from artifacts; a live adapter test exercises raw lookup plus an SDK belief fold against Ponder when the indexer is up. Next: bind execution so mined campaign transactions exist, then reconcile a real local run. +6. **[x] Bind execution to contracts and deep reconciliation.** [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) turns planned actions into contract writes, records mined hashes, and updates the public runtime-binding artifact. `npm run gen:campaign:execute` is local-first, refuses remote mutation without `--confirm-remote-mutation`, and keeps wallet secrets outside the campaign directory. Reconciliation already maps those hashes to Ponder events and SDK folds. Next: run the full 100-user campaign against the local stack, including wallet/token provisioning for generated keys beyond the funded Hardhat accounts. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. diff --git a/fake-data-generation/campaignActionAdapter.ts b/fake-data-generation/campaignActionAdapter.ts new file mode 100644 index 00000000..14bbec89 --- /dev/null +++ b/fake-data-generation/campaignActionAdapter.ts @@ -0,0 +1,240 @@ +import type { Address, Hex, PublicClient } from 'viem'; +import { parseEther, parseUnits } from 'viem'; +import { + AlignmentAttestationsAbi, + BeliefsAbi, + DelegatableNotesAbi, + ImplicationsAbi, + MutableRefUpdaterAbi, + ProjectFactoryAbi, + PublishedDataAbi, + AssuranceContractAbi, +} from '@commonality/sdk/abis'; +import { believeStatement, disbelieveStatement, attestImplication } from '@commonality/sdk/conceptspace'; +import { depositETH, delegateNote, revokeNote } from '@commonality/sdk/delegation'; +import { createDefaultDocumentStore, createDisplayableDocument, createStatement } from '@commonality/sdk/displayable-documents'; +import { attestAlignment, PROJECT_ALIGNMENT_TOPIC, toSubjectId } from '@commonality/sdk/fundingportals'; +import { buyProjectTokens, createProject, getProject } from '@commonality/sdk/lazy-giving'; +import { createSDKMachinery } from '@commonality/sdk/machinery'; +import { updateRef } from '@commonality/sdk/mutable-refs'; +import { createIPFSConfigInNodeJSFromTheUsualEnvVars } from '@commonality/sdk/node'; +import type { IpfsCidV1, WriteClients } from '@commonality/sdk/utils'; +import type { CampaignContracts, CampaignWalletBinding } from './campaignEnvironment.js'; +import type { CampaignExecutionAdapter, CampaignReceipt } from './campaignExecutor.js'; +import type { CampaignPlan, PlannedAction, PlannedProject } from './campaignPlanner.js'; +import type { CampaignRuntimeBindings } from './campaignRuntimeBindings.js'; +import { writeRuntimeBindings } from './campaignRuntimeBindings.js'; +import { buildSeedRosterDocument } from './seedCauseRoster.js'; +import { createSeedClients } from './seedRpc.js'; +import { getPaymentTokenDecimals } from './paymentTokenUnits.js'; + +const GAS_UNITS: Record = { + 'publish-statement': 180_000n, 'create-cause': 120_000n, 'set-belief': 90_000n, + 'attest-implication': 130_000n, 'create-project': 1_100_000n, 'attest-alignment': 130_000n, + 'fund-project': 180_000n, 'deposit-note': 150_000n, 'delegate-note': 100_000n, 'revoke-delegation': 90_000n, +}; + +export interface CampaignSubmittedWrite { + hash: Hex; + statementCid?: IpfsCidV1; + cause?: { owner: Address; refName: string; rosterCid: IpfsCidV1 }; + project?: { assurance: Address; token: Address }; + note?: { contractAddress: Address; noteId: string }; +} + +export interface CampaignActionWriter { + submit(action: PlannedAction, actor: WriteClients): Promise; +} + +export function classifyCampaignError(error: unknown): { retryable: boolean; category: string; message: string } { + const message = error instanceof Error ? error.message : String(error); + const retryable = /rate limit|429|timeout|ECONNRESET|nonce too low|replacement/i.test(message); + const category = /revert|execution reverted/i.test(message) ? 'contract-revert' : retryable ? 'rpc' : 'adapter'; + return { retryable, category, message }; +} + +function requireBound(value: T | undefined, label: string): T { + if (value === undefined || value === null || value === '') throw new Error(`campaign action is missing runtime binding for ${label}`); + return value; +} + +export function applySubmittedBindings(bindings: CampaignRuntimeBindings, action: PlannedAction, write: CampaignSubmittedWrite, actor: Address, now: Date): void { + bindings.updatedAt = now.toISOString(); + if (action.actorUserId) bindings.users[action.actorUserId] = actor; + if (action.statementId && write.statementCid) bindings.statements[action.statementId] = write.statementCid; + if (action.causeId && write.cause) bindings.causes[action.causeId] = write.cause; + if (action.projectId && write.project) bindings.projects[action.projectId] = write.project.assurance; + if (action.noteId && write.note) bindings.notes[action.noteId] = write.note; +} + +export function createCampaignContractAdapter(input: { + plan: CampaignPlan; + contracts: CampaignContracts; + wallets: readonly CampaignWalletBinding[]; + publisher: CampaignWalletBinding; + bindings: CampaignRuntimeBindings; + writer: CampaignActionWriter; + getReceipt: (hash: Hex) => Promise; + persistBindings?: (bindings: CampaignRuntimeBindings) => Promise; + clientsFor?: (wallet: CampaignWalletBinding) => WriteClients; + gasPrice?: bigint; + now?: () => Date; +}): CampaignExecutionAdapter { + const wallets = new Map(input.wallets.map((wallet) => [wallet.walletSlot, wallet])); + const users = new Map(input.plan.users.map((user) => [user.id, user])); + const projectTokens = new Map(); + const gasPrice = input.gasPrice ?? 1_000_000_000n; + const now = input.now ?? (() => new Date()); + const clientsFor = input.clientsFor ?? ((wallet: CampaignWalletBinding) => createSeedClients(wallet.privateKey) as WriteClients); + + const actorFor = (action: PlannedAction): CampaignWalletBinding => { + if (!action.actorUserId) return input.publisher; + const user = requireBound(users.get(action.actorUserId), `user ${action.actorUserId}`); + return requireBound(wallets.get(user.walletSlot), `wallet ${user.walletSlot}`); + }; + + return { + estimateNativeCost: async (action) => GAS_UNITS[action.type] * gasPrice, + classifyError: classifyCampaignError, + getReceipt: input.getReceipt, + async submit(action) { + const wallet = actorFor(action); + const clients = clientsFor(wallet); + const write = await input.writer.submit(action, clients); + applySubmittedBindings(input.bindings, action, write, clients.account, now()); + if (write.project) projectTokens.set(action.projectId!, write.project.token); + await input.persistBindings?.(input.bindings); + return write.hash; + }, + }; +} + +export function createLiveCampaignActionWriter(input: { + plan: CampaignPlan; + contracts: CampaignContracts; + bindings: CampaignRuntimeBindings; + projectTokens?: Map; +}): CampaignActionWriter { + const machinery = createSDKMachinery({ + ipfsConfig: createIPFSConfigInNodeJSFromTheUsualEnvVars(), + eventCacheUrl: process.env.EVENT_CACHE_URL ?? 'http://localhost:42069', + }); + const statements = new Map(input.plan.statements.map((item) => [item.id, item])); + const projects = new Map(input.plan.projects.map((item) => [item.id, item])); + const projectTokens = input.projectTokens ?? new Map(); + const notesContract = { address: input.contracts.delegatableNotes, abi: DelegatableNotesAbi }; + const decimals = getPaymentTokenDecimals(); + + const statementCid = (id: string | undefined): IpfsCidV1 => requireBound(input.bindings.statements[requireBound(id, 'statement id')], `statement CID ${id}`); + const projectAddress = (id: string | undefined): Address => requireBound(input.bindings.projects[requireBound(id, 'project id')], `project ${id}`); + const storeFor = (clients: WriteClients) => createDefaultDocumentStore(machinery, { + clients, + publishedDataContract: { address: input.contracts.publishedData, abi: PublishedDataAbi }, + }); + const noteAmount = (action: PlannedAction) => parseEther((Math.max(1, action.amount ?? 1) / 100_000).toString()); + + const handlers: Record Promise> = { + async 'publish-statement'(action, clients) { + const planned = requireBound(statements.get(action.statementId!), `statement ${action.statementId}`); + const publication = await storeFor(clients).publish(createStatement({ + content: planned.text, topic: planned.causeId, extras: { campaign: input.plan.campaignId, synthetic: true }, + })); + return { hash: publication.txHash, statementCid: publication.cid }; + }, + async 'create-cause'(action, clients) { + const plankCids = input.plan.statements.filter((item) => item.causeId === action.causeId).map((item) => statementCid(item.id)); + const title = action.causeId ?? 'campaign-cause'; + const rosterCid = (await storeFor(clients).publish(buildSeedRosterDocument({ + title, summary: `SYNTHETIC TESTNET CAMPAIGN cause ${title}`, plankCids, mediatorBlurb: '', + }))).cid; + const refName = `campaign-${input.plan.campaignId}-${action.causeId}`; + const hash = await updateRef(clients, { address: input.contracts.mutableRefUpdater, abi: MutableRefUpdaterAbi }, refName, rosterCid); + return { hash, cause: { owner: clients.account, refName, rosterCid } }; + }, + async 'set-belief'(action, clients) { + const beliefs = { address: input.contracts.beliefs, abi: BeliefsAbi }; + const cid = statementCid(action.statementId); + const hash = action.belief === 'disbelieve' ? await disbelieveStatement(clients, beliefs, cid) : await believeStatement(clients, beliefs, cid); + return { hash }; + }, + async 'attest-implication'(action, clients) { + return { hash: await attestImplication(clients, { address: input.contracts.implications, abi: ImplicationsAbi }, statementCid(action.implication!.fromStatementId), statementCid(action.implication!.toStatementId)) }; + }, + async 'create-project'(action, clients) { + const planned = requireBound(projects.get(action.projectId!), `project ${action.projectId}`) as PlannedProject; + const publication = await storeFor(clients).publish(createDisplayableDocument({ + format: 'markdown-restricted', content: planned.outcome, + extras: { statementType: 'lazy-giving-project-metadata', name: planned.title, description: planned.outcome, campaign: input.plan.campaignId, synthetic: true, alignedStatementRefs: planned.statementIds }, + })); + const latest = await clients.publicClient.getBlock(); + const { hash, projectDetails } = await createProject(clients, { address: input.contracts.projectFactory, abi: ProjectFactoryAbi }, { + metadataURI: `ipfs://${publication.cid}/`, contractURI: `ipfs://${publication.cid}`, owner: clients.account, recipient: clients.account, + paymentToken: input.contracts.paymentToken, threshold: parseUnits('2', decimals), deadline: latest.timestamp + 30n * 24n * 60n * 60n, + projectMetadataCid: publication.cid, tokenIds: [1n, 2n, 3n], tokenCounts: [100n, 500n, 1000n], + tokenPrices: [parseUnits('0.1', decimals), parseUnits('0.05', decimals), parseUnits('0.01', decimals)], + }); + projectTokens.set(planned.id, projectDetails.tokenAddress); + return { hash, project: { assurance: projectDetails.assuranceContractAddress, token: projectDetails.tokenAddress } }; + }, + async 'attest-alignment'(action, clients) { + return { hash: await attestAlignment(clients, { address: input.contracts.alignmentAttestations, abi: AlignmentAttestationsAbi }, toSubjectId(projectAddress(action.projectId)), statementCid(action.statementId), PROJECT_ALIGNMENT_TOPIC) }; + }, + async 'fund-project'(action, clients) { + const assurance = projectAddress(action.projectId); + let token = projectTokens.get(action.projectId!); + if (!token) { + const folded = await getProject(machinery, assurance); + if (!folded?.erc1155Address) throw new Error(`cannot fund unknown project ${action.projectId}`); + token = folded.erc1155Address as Address; + projectTokens.set(action.projectId!, token); + } + return { hash: await buyProjectTokens(clients, { address: assurance, abi: AssuranceContractAbi }, { buyer: clients.account, tokenAddress: token, tokenIds: [3n], tokenCounts: [1n], totalCost: parseUnits('0.01', decimals) }) }; + }, + async 'deposit-note'(action, clients) { + const { hash, noteId } = await depositETH(clients, notesContract, { amount: noteAmount(action) }); + return { hash, note: { contractAddress: input.contracts.delegatableNotes, noteId: noteId.toString() } }; + }, + async 'delegate-note'(action, clients) { + const binding = requireBound(input.bindings.notes[action.noteId!], `note ${action.noteId}`); + const { hash, delegatedNoteId } = await delegateNote(clients, notesContract, { + noteId: BigInt(binding.noteId), owners: [clients.account], + delegateTo: requireBound(input.bindings.users[action.delegateUserId!], `delegate ${action.delegateUserId}`), + amount: noteAmount(action), + }); + return { hash, note: { ...binding, noteId: delegatedNoteId.toString() } }; + }, + async 'revoke-delegation'(action, clients) { + const binding = requireBound(input.bindings.notes[action.noteId!], `note ${action.noteId}`); + const delegated = input.plan.actions.find((item) => item.type === 'delegate-note' && item.noteId === action.noteId); + const hash = await revokeNote(clients, notesContract, { + noteId: BigInt(binding.noteId), + owners: [requireBound(input.bindings.users[delegated?.delegateUserId ?? ''], `delegate for ${action.noteId}`), clients.account], + }); + return { hash, note: binding }; + }, + }; + + return { submit: (action, clients) => handlers[action.type](action, clients) }; +} + +export async function persistCampaignBindings(plan: CampaignPlan, bindings: CampaignRuntimeBindings, outputPath: string): Promise { + await writeRuntimeBindings(plan, bindings, outputPath); +} + +export function createReceiptLookup(publicClient: Pick): (hash: Hex) => Promise { + return async (hash) => { + try { + const receipt = await publicClient.getTransactionReceipt({ hash }); + if (!receipt) return null; + return { + status: receipt.status === 'success' ? 'success' : 'reverted', + gasUsed: receipt.gasUsed, + effectiveGasPrice: receipt.effectiveGasPrice ?? 0n, + blockNumber: receipt.blockNumber, + }; + } catch { + return null; + } + }; +} + diff --git a/fake-data-generation/campaignExecute.ts b/fake-data-generation/campaignExecute.ts new file mode 100644 index 00000000..a6db639f --- /dev/null +++ b/fake-data-generation/campaignExecute.ts @@ -0,0 +1,126 @@ +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { generatePrivateKey, privateKeyToAccount } from 'viem/accounts'; +import { + LOCAL_HARDHAT_CHAIN_ID, + loadCampaignEnvironment, + preflightCampaignEnvironment, + createCampaignChainAdapter, + validateCampaignWallets, + type CampaignWalletBinding, +} from './campaignEnvironment.js'; +import { createCampaignContractAdapter, createLiveCampaignActionWriter, createReceiptLookup, persistCampaignBindings } from './campaignActionAdapter.js'; +import { executeCampaignPlan } from './campaignExecutor.js'; +import { loadCampaignPlan } from './campaignPlanner.js'; +import { createEmptyRuntimeBindings, loadRuntimeBindings } from './campaignRuntimeBindings.js'; +import type { CampaignManifestV1 } from './campaignSchema.js'; +import { FUNDED_HARDHAT_DEV_KEYS } from './seedCauseRoster.js'; +import { createSeedPublicClient } from './seedRpc.js'; +import { loadEnv, RPC_URL } from './loadEnv.js'; +import { HARDHAT_PRIVATE_KEYS } from './generateUsers.js'; + +loadEnv(); + +function parseFlag(name: string): boolean { + return process.argv.includes(name); +} + +function parseOption(name: string, fallback?: string): string | undefined { + const index = process.argv.indexOf(name); + if (index >= 0 && process.argv[index + 1]) return process.argv[index + 1]; + return fallback; +} + +async function loadOrCreateWallets(planUsers: { id: string; walletSlot: string }[], secretsPath: string, local: boolean): Promise { + try { + const saved = JSON.parse(await readFile(secretsPath, 'utf8')) as CampaignWalletBinding[]; + return saved; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + const wallets = planUsers.map((user, index) => { + const privateKey = local && index < FUNDED_HARDHAT_DEV_KEYS.length + ? FUNDED_HARDHAT_DEV_KEYS[index] as `0x${string}` + : generatePrivateKey(); + const account = privateKeyToAccount(privateKey); + return { + walletSlot: user.walletSlot, + address: account.address, + privateKey, + source: (local && index < FUNDED_HARDHAT_DEV_KEYS.length ? 'hardhat' : 'generated') as CampaignWalletBinding['source'], + }; + }); + await mkdir(path.dirname(secretsPath), { recursive: true }); + await writeFile(secretsPath, `${JSON.stringify(wallets, null, 2)}\n`); + return wallets; +} + +async function main(): Promise { + const directory = path.dirname(fileURLToPath(import.meta.url)); + const mode = parseOption('--mode', 'local') as 'local' | 'remote'; + const manifestPath = parseOption('--manifest', path.join(directory, 'campaigns/medium-realistic-v1.json'))!; + const outputDirectory = parseOption('--output', path.join(directory, 'output/campaigns/medium-realistic-v1'))!; + const deploymentEnvPath = parseOption('--deployment-env', path.join(directory, '../deployments/localhost.env'))!; + const mutationConfirmed = parseFlag('--confirm-remote-mutation'); + const manifest = JSON.parse(await readFile(manifestPath, 'utf8')) as CampaignManifestV1; + const plan = await loadCampaignPlan(manifest, outputDirectory); + const environment = await loadCampaignEnvironment({ + mode, + rpcUrl: RPC_URL, + expectedChainId: mode === 'local' ? LOCAL_HARDHAT_CHAIN_ID : Number(parseOption('--chain-id', '0')), + deploymentEnvPath, + mutationConfirmed, + }); + if (environment.mode === 'remote' && !environment.mutationConfirmed) { + throw new Error('remote campaign execution requires --confirm-remote-mutation'); + } + await preflightCampaignEnvironment(environment, createCampaignChainAdapter(environment.rpcUrl)); + const secretsPath = path.join(outputDirectory, manifest.artifactLayout.walletSecrets); + const wallets = await loadOrCreateWallets(plan.users, secretsPath, environment.mode === 'local'); + validateCampaignWallets(environment, wallets, HARDHAT_PRIVATE_KEYS); + const publisher = wallets[0]; + const bindingsPath = path.join(outputDirectory, manifest.artifactLayout.runtimeBindings ?? 'execution/runtime-bindings.json'); + let bindings; + try { + bindings = await loadRuntimeBindings(plan, bindingsPath); + } catch { + bindings = createEmptyRuntimeBindings(plan); + } + for (const user of plan.users) { + const wallet = wallets.find((item) => item.walletSlot === user.walletSlot); + if (wallet) bindings.users[user.id] = wallet.address; + } + await persistCampaignBindings(plan, bindings, bindingsPath); + const publicClient = createSeedPublicClient(environment.rpcUrl); + const adapter = createCampaignContractAdapter({ + plan, + contracts: environment.contracts, + wallets, + publisher, + bindings, + writer: createLiveCampaignActionWriter({ plan, contracts: environment.contracts, bindings }), + getReceipt: createReceiptLookup(publicClient), + persistBindings: (value) => persistCampaignBindings(plan, value, bindingsPath), + }); + const summary = await executeCampaignPlan({ + campaignId: plan.campaignId, + manifestFingerprint: plan.manifestFingerprint, + actions: plan.actions, + adapter, + options: { + statePath: path.join(outputDirectory, manifest.artifactLayout.executionState), + concurrency: Number(parseOption('--concurrency', '1')), + pacingMs: Number(parseOption('--pacing-ms', environment.mode === 'local' ? '0' : '250')), + maxRetries: Number(parseOption('--max-retries', '2')), + retryBackoffMs: Number(parseOption('--retry-backoff-ms', '500')), + transactionCap: Number(parseOption('--transaction-cap', String(plan.actions.length))), + nativeTokenBudget: BigInt(parseOption('--native-budget-wei', '10000000000000000000')!), + }, + }); + console.log(`Campaign ${plan.campaignId}: mined ${summary.mined}, failed ${summary.failed}, submitted ${summary.submitted}, planned ${summary.planned}.`); +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main().catch((error) => { console.error(error instanceof Error ? error.message : error); process.exitCode = 1; }); +} diff --git a/fake-data-generation/campaignSchema.ts b/fake-data-generation/campaignSchema.ts index d5d5ec6a..bca18dfe 100644 --- a/fake-data-generation/campaignSchema.ts +++ b/fake-data-generation/campaignSchema.ts @@ -125,7 +125,7 @@ function validateActionRules(manifest: CampaignManifestV1): void { } function validateArtifactLayout(manifest: CampaignManifestV1): void { - const requiredArtifacts = ['manifest', 'statementCatalog', 'assignments', 'walletAddresses', 'walletSecrets', 'actionPlan', 'executionState', 'fundingLedger', 'reconciliation', 'browserObservations', 'summary']; + const requiredArtifacts = ['manifest', 'statementCatalog', 'assignments', 'walletAddresses', 'walletSecrets', 'actionPlan', 'executionState', 'runtimeBindings', 'fundingLedger', 'reconciliation', 'browserObservations', 'summary']; for (const artifact of requiredArtifacts) { if (!manifest.artifactLayout[artifact]) throw new Error(`artifactLayout.${artifact} is required`); } diff --git a/fake-data-generation/campaigns/medium-realistic-v1.json b/fake-data-generation/campaigns/medium-realistic-v1.json index 072d2ccb..a174c0a6 100644 --- a/fake-data-generation/campaigns/medium-realistic-v1.json +++ b/fake-data-generation/campaigns/medium-realistic-v1.json @@ -110,6 +110,7 @@ "walletSecrets": "../secrets/medium-realistic-v1.wallets.json", "actionPlan": "plan/actions.json", "executionState": "execution/actions.jsonl", + "runtimeBindings": "execution/runtime-bindings.json", "fundingLedger": "execution/funding-ledger.json", "reconciliation": "reports/reconciliation.json", "browserObservations": "reports/browser-observations.json", diff --git a/fake-data-generation/package.json b/fake-data-generation/package.json index 82b7d968..68acf0e6 100644 --- a/fake-data-generation/package.json +++ b/fake-data-generation/package.json @@ -18,6 +18,7 @@ "test:seed:implication-regression": "tsx verifySeedImplicationEvaluations.ts", "gen:proliferation": "tsx generateProliferation.ts", "gen:campaign:plan": "tsx campaignPlanner.ts", + "gen:campaign:execute": "tsx campaignExecute.ts", "gen:campaign:reconcile": "tsx campaignReconcile.ts", "gen:attesters": "tsx generateAttesters.ts", "gen:attestations": "tsx generateAttestations.ts", diff --git a/fake-data-generation/test/campaignActionAdapter.test.ts b/fake-data-generation/test/campaignActionAdapter.test.ts new file mode 100644 index 00000000..ae2dd43c --- /dev/null +++ b/fake-data-generation/test/campaignActionAdapter.test.ts @@ -0,0 +1,110 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import type { Hex } from 'viem'; +import { privateKeyToAccount } from 'viem/accounts'; +import { fakeIpfsCidV1 } from '@commonality/sdk/testing'; +import { + applySubmittedBindings, + classifyCampaignError, + createCampaignContractAdapter, + type CampaignActionWriter, +} from '../campaignActionAdapter.js'; +import { executeCampaignPlan } from '../campaignExecutor.js'; +import { createEmptyRuntimeBindings, loadRuntimeBindings } from '../campaignRuntimeBindings.js'; +import type { CampaignPlan, PlannedAction } from '../campaignPlanner.js'; +import type { CampaignWalletBinding } from '../campaignEnvironment.js'; +import type { WriteClients } from '@commonality/sdk/utils'; + +const publisherKey = '0x59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d' as const; +const userKey = '0x5de4111afa1a4b94908f83103eb1f1706367c2e68ca870fc3fb9a804cdab365a' as const; +const publisher = privateKeyToAccount(publisherKey); +const user = privateKeyToAccount(userKey); + +const actions: PlannedAction[] = [ + { id: 'action-1', sequence: 1, type: 'publish-statement', actorUserId: null, statementId: 'statement-1', causeId: 'open-source', dependsOn: [] }, + { id: 'action-2', sequence: 2, type: 'create-cause', actorUserId: 'user-1', causeId: 'open-source', dependsOn: ['action-1'] }, + { id: 'action-3', sequence: 3, type: 'set-belief', actorUserId: 'user-1', statementId: 'statement-1', belief: 'believe', dependsOn: ['action-1'] }, +]; + +const plan = { + version: 'commonality-campaign-plan-v1', campaignId: 'test', deterministicSeed: 'seed', manifestFingerprint: 'fingerprint', + users: [{ id: 'user-1', walletSlot: 'wallet-user-1' }], + statements: [{ id: 'statement-1', causeId: 'open-source' }], + projects: [], + actions, +} as unknown as CampaignPlan; + +const wallets: CampaignWalletBinding[] = [ + { walletSlot: 'wallet-user-1', address: user.address, privateKey: userKey, source: 'generated' }, +]; +const publisherWallet: CampaignWalletBinding = { walletSlot: 'publisher', address: publisher.address, privateKey: publisherKey, source: 'generated' }; + +function writer(calls: string[]): CampaignActionWriter { + return { + async submit(action) { + calls.push(action.type); + const hash = `0x${action.sequence.toString(16).padStart(64, '0')}` as Hex; + if (action.type === 'publish-statement') return { hash, statementCid: fakeIpfsCidV1(action.statementId!) }; + if (action.type === 'create-cause') { + return { hash, cause: { owner: user.address, refName: 'cause-open-source', rosterCid: fakeIpfsCidV1('roster') } }; + } + return { hash }; + }, + }; +} + +test('contract adapter records mined hashes and public runtime bindings', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-adapter-')); + const calls: string[] = []; + const bindings = createEmptyRuntimeBindings(plan); + bindings.users['user-1'] = user.address; + try { + const adapter = createCampaignContractAdapter({ + plan, + contracts: { + beliefs: user.address, implications: user.address, alignmentAttestations: user.address, + delegatableNotes: user.address, projectFactory: user.address, paymentToken: user.address, + publishedData: user.address, mutableRefUpdater: user.address, + }, + wallets, + publisher: publisherWallet, + bindings, + writer: writer(calls), + getReceipt: async () => ({ status: 'success', gasUsed: 21_000n, effectiveGasPrice: 1n, blockNumber: 8n }), + persistBindings: async (value) => { + const { writeRuntimeBindings } = await import('../campaignRuntimeBindings.js'); + await writeRuntimeBindings(plan, value, path.join(directory, 'bindings.json')); + }, + clientsFor: (wallet) => ({ account: wallet.address } as WriteClients), + }); + const summary = await executeCampaignPlan({ + campaignId: plan.campaignId, + manifestFingerprint: plan.manifestFingerprint, + actions, + adapter, + options: { statePath: path.join(directory, 'execution.json'), concurrency: 1, pacingMs: 0, maxRetries: 0, retryBackoffMs: 0, transactionCap: 10, nativeTokenBudget: 10n ** 18n }, + }); + assert.equal(summary.mined, 3); + assert.deepEqual(calls, ['publish-statement', 'create-cause', 'set-belief']); + const stored = await loadRuntimeBindings(plan, path.join(directory, 'bindings.json')); + assert.equal(stored.statements['statement-1'], fakeIpfsCidV1('statement-1')); + assert.equal(stored.causes['open-source'].refName, 'cause-open-source'); + assert.equal(stored.users['user-1'], user.address); + const execution = JSON.parse(await readFile(path.join(directory, 'execution.json'), 'utf8')) as { actions: Array<{ transactionHash: string }> }; + assert.equal(execution.actions[0].transactionHash, `0x${'1'.padStart(64, '0')}`); + } finally { await rm(directory, { recursive: true, force: true }); } +}); + +test('classifyCampaignError treats rate limits as retryable RPC failures', () => { + assert.deepEqual(classifyCampaignError(new Error('429 rate limit')), { retryable: true, category: 'rpc', message: '429 rate limit' }); + assert.equal(classifyCampaignError(new Error('execution reverted: nope')).retryable, false); +}); + +test('applySubmittedBindings never writes private keys', () => { + const bindings = createEmptyRuntimeBindings(plan); + applySubmittedBindings(bindings, actions[0], { hash: `0x${'2'.repeat(64)}` as Hex, statementCid: fakeIpfsCidV1('statement-1') }, publisher.address, new Date('2026-01-01T00:00:00.000Z')); + assert.doesNotMatch(JSON.stringify(bindings), /privateKey|0x5de4111afa1a4b94908f83103eb1f1706367c2e68ca870fc3fb9a804cdab365a/); +}); From bc1c2255a798f184529c6efa55a118b326bbcdae Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 15:28:31 -0400 Subject: [PATCH 13/19] Fund generated campaign wallets before local execution. Local mode transfers ETH and payment tokens (with mint fallback) so the 100-user campaign is not limited to the ten funded Hardhat accounts. Remote stays transfer-only. --- CONTINUITY.md | 4 + fake-data-generation/README.md | 4 +- .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignExecute.ts | 20 ++ fake-data-generation/campaignProvisioning.ts | 188 ++++++++++++++++++ .../test/campaignProvisioning.test.ts | 110 ++++++++++ 6 files changed, 325 insertions(+), 3 deletions(-) create mode 100644 fake-data-generation/campaignProvisioning.ts create mode 100644 fake-data-generation/test/campaignProvisioning.test.ts diff --git a/CONTINUITY.md b/CONTINUITY.md index 0635636e..f4180b1f 100644 --- a/CONTINUITY.md +++ b/CONTINUITY.md @@ -2151,3 +2151,7 @@ beneficiary lookup and bulk/bot UX. ## 2026-09-11 — Campaign reconciliation talks to the local stack Item 6 of the medium-realistic campaign: indexer adapter paginates with `fetchEventsComplete` and collapses alternative proving events so ordinary vs retroactive funding is one write. `npm run gen:campaign:reconcile -- --probe` reads live chain/indexer heads. Without `--probe` it reconciles from plan + execution + runtime-bindings. A live adapter test exercises raw lookup plus an SDK belief fold against Ponder. Campaign execution adapter is still required before mined campaign txs exist to close item 6. + +## 2026-09-11 — Local campaign wallet provisioning + +Item 7 of the medium-realistic campaign: `campaignProvisioning.ts` funds generated wallets before `gen:campaign:execute`. Local mode transfers ETH from the Hardhat funder and payment tokens (mint fallback). Remote stays transfer-only and refuses Hardhat keys. `--skip-provision` exists for already-funded reruns. Live smoke on the local stack wrote `execution/funding-ledger.json` for all 100 slots. Full 100-user execute/reconcile/UI inspection is still open. diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 1c3c33ad..693ab934 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -15,9 +15,9 @@ The medium realistic campaign has a separate planning-only command: `npm run gen Campaign execution uses the explicit environment boundary in `campaignEnvironment.ts`. Local mode is pinned to Hardhat chain 31337 and may use the legacy deploy/Hardhat-wallet/free-mint conveniences. Remote mode requires a committed deployment env manifest, an explicit non-Hardhat chain ID, existing bytecode at every required address, generated campaign wallets, and transfer-only payment-token provisioning. Deployment and provisioning adapters are validated against those policies before they can run; remote execution must also carry a separate mutation-confirmation flag when it is added. The existing `gen:tiny`/`gen:small`/`gen:medium`/`gen:large` commands remain local-only legacy simulation paths and are not remote campaign commands. -The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) is the contract-binding layer: it maps each planned action type onto SDK writes, classifies RPC vs revert failures, and updates public runtime bindings (no private keys). `npm run gen:campaign:execute` runs that adapter locally after environment preflight; remote mode still requires `--confirm-remote-mutation`. Generated wallet secrets stay under `output/campaigns/secrets/` (gitignored via `output/`). +The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) is the contract-binding layer: it maps each planned action type onto SDK writes, classifies RPC vs revert failures, and updates public runtime bindings (no private keys). `npm run gen:campaign:execute` runs that adapter locally after environment preflight and funds generated wallets (ETH plus payment tokens, minting locally when transfer fails). Remote mode still requires `--confirm-remote-mutation` and stays transfer-only. Pass `--skip-provision` only when wallets are already funded. Generated wallet secrets stay under `output/campaigns/secrets/` (gitignored via `output/`). The funding ledger is `execution/funding-ledger.json`. -`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. `npm run gen:campaign:reconcile -- --probe` talks to the live local chain and Ponder `/status` heads. The same command without `--probe` loads planned actions, execution state, and runtime bindings, then writes the reconciliation JSON/summary through the real indexer adapter and SDK folds. Raw lookups paginate with `fetchEventsComplete` and treat ordinary vs retroactive funding events as one proving write. A live adapter test runs when `http://localhost:42069` is reachable. Full 100-user local execution still needs provisioning for wallets beyond the funded Hardhat accounts; a dry adapter test covers binding updates without a live chain. +`campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. `npm run gen:campaign:reconcile -- --probe` talks to the live local chain and Ponder `/status` heads. The same command without `--probe` loads planned actions, execution state, and runtime bindings, then writes the reconciliation JSON/summary through the real indexer adapter and SDK folds. Raw lookups paginate with `fetchEventsComplete` and treat ordinary vs retroactive funding events as one proving write. A live adapter test runs when `http://localhost:42069` is reachable. Full 100-user local execution provisions wallets beyond the funded Hardhat accounts before submitting writes; a dry adapter test covers binding updates without a live chain. ## Overview diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 07ae83ec..f4d3532e 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -182,7 +182,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 3. **[x] Split local assumptions from reusable execution.** [`campaignEnvironment.ts`](./campaignEnvironment.ts) defines explicit local/remote chain configuration plus deployment, provisioning, wallet, and read-only chain adapters. Local mode is pinned to chain 31337 and retains legacy conveniences; remote mode requires an existing deployment manifest, generated non-Hardhat wallets, transfer-only token provisioning, an explicit mutation-confirmation field, the expected non-local chain ID, and bytecode at every required contract. Adapter and preflight tests cover wrong chains, absent code, implicit deploy/mint policies, key/address mismatches, and Hardhat-key rejection. Existing tiny/demo/local commands remain unchanged and local-only. 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. -6. **[x] Bind execution to contracts and deep reconciliation.** [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) turns planned actions into contract writes, records mined hashes, and updates the public runtime-binding artifact. `npm run gen:campaign:execute` is local-first, refuses remote mutation without `--confirm-remote-mutation`, and keeps wallet secrets outside the campaign directory. Reconciliation already maps those hashes to Ponder events and SDK folds. Next: run the full 100-user campaign against the local stack, including wallet/token provisioning for generated keys beyond the funded Hardhat accounts. +6. **[x] Bind execution to contracts and deep reconciliation.** [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) turns planned actions into contract writes, records mined hashes, and updates the public runtime-binding artifact. `npm run gen:campaign:execute` is local-first, refuses remote mutation without `--confirm-remote-mutation`, and keeps wallet secrets outside the campaign directory. Reconciliation already maps those hashes to Ponder events and SDK folds. Local execution now funds generated wallets (ETH plus payment tokens, with mint fallback) and writes `execution/funding-ledger.json` before submitting writes. 7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. diff --git a/fake-data-generation/campaignExecute.ts b/fake-data-generation/campaignExecute.ts index a6db639f..719d3d11 100644 --- a/fake-data-generation/campaignExecute.ts +++ b/fake-data-generation/campaignExecute.ts @@ -13,6 +13,7 @@ import { import { createCampaignContractAdapter, createLiveCampaignActionWriter, createReceiptLookup, persistCampaignBindings } from './campaignActionAdapter.js'; import { executeCampaignPlan } from './campaignExecutor.js'; import { loadCampaignPlan } from './campaignPlanner.js'; +import { createLiveCampaignFundingChain, provisionCampaignWallets } from './campaignProvisioning.js'; import { createEmptyRuntimeBindings, loadRuntimeBindings } from './campaignRuntimeBindings.js'; import type { CampaignManifestV1 } from './campaignSchema.js'; import { FUNDED_HARDHAT_DEV_KEYS } from './seedCauseRoster.js'; @@ -79,6 +80,25 @@ async function main(): Promise { const secretsPath = path.join(outputDirectory, manifest.artifactLayout.walletSecrets); const wallets = await loadOrCreateWallets(plan.users, secretsPath, environment.mode === 'local'); validateCampaignWallets(environment, wallets, HARDHAT_PRIVATE_KEYS); + await writeFile(path.join(outputDirectory, manifest.artifactLayout.walletAddresses), `${JSON.stringify({ + version: plan.version, campaignId: plan.campaignId, + wallets: plan.users.map((user) => { + const wallet = wallets.find((item) => item.walletSlot === user.walletSlot); + return { userId: user.id, walletSlot: user.walletSlot, address: wallet?.address ?? null, status: wallet ? 'provisioned' : 'unprovisioned' }; + }), + }, null, 2)}\n`); + const funder = wallets.find((wallet) => wallet.source === 'hardhat') ?? wallets[0]; + if (!funder) throw new Error('campaign has no funder wallet'); + const skipProvision = parseFlag('--skip-provision'); + if (!skipProvision) { + await provisionCampaignWallets({ + environment, + plan, + wallets, + chain: createLiveCampaignFundingChain({ funderPrivateKey: funder.privateKey, contracts: environment.contracts }), + ledgerPath: path.join(outputDirectory, manifest.artifactLayout.fundingLedger), + }); + } const publisher = wallets[0]; const bindingsPath = path.join(outputDirectory, manifest.artifactLayout.runtimeBindings ?? 'execution/runtime-bindings.json'); let bindings; diff --git a/fake-data-generation/campaignProvisioning.ts b/fake-data-generation/campaignProvisioning.ts new file mode 100644 index 00000000..981f896a --- /dev/null +++ b/fake-data-generation/campaignProvisioning.ts @@ -0,0 +1,188 @@ +import { mkdir, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { parseEther, parseUnits, type Address, type Hex } from 'viem'; +import type { CampaignContracts, CampaignEnvironment, CampaignWalletBinding } from './campaignEnvironment.js'; +import type { CampaignPlan, PlannedAction } from './campaignPlanner.js'; +import { getPaymentTokenDecimals } from './paymentTokenUnits.js'; +import { createSeedClients } from './seedRpc.js'; + +const GAS_UNITS: Record = { + 'publish-statement': 180_000n, 'create-cause': 120_000n, 'set-belief': 90_000n, + 'attest-implication': 130_000n, 'create-project': 1_100_000n, 'attest-alignment': 130_000n, + 'fund-project': 180_000n, 'deposit-note': 150_000n, 'delegate-note': 100_000n, 'revoke-delegation': 90_000n, +}; + +const DEFAULT_GAS_PRICE = 1_000_000_000n; +const NATIVE_BUFFER_WEI = parseEther('0.05'); +const FUND_PROJECT_TOKEN = '0.01'; + +export const PAYMENT_TOKEN_FUNDING_ABI = [ + { + name: 'balanceOf', + type: 'function', + stateMutability: 'view', + inputs: [{ name: 'account', type: 'address' }], + outputs: [{ name: '', type: 'uint256' }], + }, + { + name: 'transfer', + type: 'function', + stateMutability: 'nonpayable', + inputs: [ + { name: 'to', type: 'address' }, + { name: 'amount', type: 'uint256' }, + ], + outputs: [{ name: '', type: 'bool' }], + }, + { + name: 'mintTo', + type: 'function', + stateMutability: 'nonpayable', + inputs: [ + { name: 'to', type: 'address' }, + { name: 'amount', type: 'uint256' }, + ], + outputs: [], + }, +] as const; + +export interface CampaignWalletNeed { + walletSlot: string; + address: Address; + nativeWei: bigint; + paymentTokenUnits: bigint; +} + +export interface CampaignFundingLedger { + campaignId: string; + mode: CampaignEnvironment['mode']; + paymentTokenStrategy: CampaignEnvironment['provisioning']['paymentTokenStrategy']; + wallets: Array<{ + walletSlot: string; + address: Address; + nativeWei: string; + paymentTokenUnits: string; + nativeTransferred: boolean; + paymentTokenMethod: 'none' | 'skipped' | 'transfer' | 'mint'; + }>; +} + +export interface CampaignFundingChain { + getNativeBalance(address: Address): Promise; + getTokenBalance(address: Address): Promise; + transferNative(to: Address, amount: bigint): Promise; + transferToken(to: Address, amount: bigint): Promise; + mintToken?(to: Address, amount: bigint): Promise; +} + +function noteDepositWei(action: PlannedAction): bigint { + return parseEther((Math.max(1, action.amount ?? 1) / 100_000).toString()); +} + +export function computeCampaignFundingNeeds(plan: CampaignPlan, wallets: readonly CampaignWalletBinding[], gasPrice = DEFAULT_GAS_PRICE): CampaignWalletNeed[] { + const users = new Map(plan.users.map((user) => [user.id, user])); + const bySlot = new Map(); + for (const wallet of wallets) { + bySlot.set(wallet.walletSlot, { walletSlot: wallet.walletSlot, address: wallet.address, nativeWei: NATIVE_BUFFER_WEI, paymentTokenUnits: 0n }); + } + const decimals = getPaymentTokenDecimals(); + const fundCost = parseUnits(FUND_PROJECT_TOKEN, decimals); + for (const action of plan.actions) { + const user = action.actorUserId ? users.get(action.actorUserId) : undefined; + if (!user) continue; + const need = bySlot.get(user.walletSlot); + if (!need) continue; + need.nativeWei += GAS_UNITS[action.type] * gasPrice; + if (action.type === 'deposit-note') need.nativeWei += noteDepositWei(action); + if (action.type === 'fund-project') need.paymentTokenUnits += fundCost; + } + return [...bySlot.values()]; +} + +export async function provisionCampaignWallets(input: { + environment: CampaignEnvironment; + plan: CampaignPlan; + wallets: readonly CampaignWalletBinding[]; + chain: CampaignFundingChain; + ledgerPath: string; +}): Promise { + if (input.environment.mode === 'remote' && input.wallets.some((wallet) => wallet.source === 'hardhat')) { + throw new Error('remote campaign provisioning refuses Hardhat wallets'); + } + const needs = computeCampaignFundingNeeds(input.plan, input.wallets); + const ledger: CampaignFundingLedger = { + campaignId: input.plan.campaignId, + mode: input.environment.mode, + paymentTokenStrategy: input.environment.provisioning.paymentTokenStrategy, + wallets: [], + }; + for (const need of needs) { + const nativeBalance = await input.chain.getNativeBalance(need.address); + let nativeTransferred = false; + if (nativeBalance < need.nativeWei) { + await input.chain.transferNative(need.address, need.nativeWei - nativeBalance); + nativeTransferred = true; + } + let paymentTokenMethod: CampaignFundingLedger['wallets'][number]['paymentTokenMethod'] = 'none'; + if (need.paymentTokenUnits > 0n) { + const tokenBalance = await input.chain.getTokenBalance(need.address); + if (tokenBalance >= need.paymentTokenUnits) { + paymentTokenMethod = 'skipped'; + } else { + const deficit = need.paymentTokenUnits - tokenBalance; + try { + await input.chain.transferToken(need.address, deficit); + paymentTokenMethod = 'transfer'; + } catch (error) { + if (input.environment.provisioning.paymentTokenStrategy !== 'transfer-or-mint' || !input.chain.mintToken) { + throw error; + } + await input.chain.mintToken(need.address, deficit); + paymentTokenMethod = 'mint'; + } + } + } + ledger.wallets.push({ + walletSlot: need.walletSlot, + address: need.address, + nativeWei: need.nativeWei.toString(), + paymentTokenUnits: need.paymentTokenUnits.toString(), + nativeTransferred, + paymentTokenMethod, + }); + } + await mkdir(path.dirname(input.ledgerPath), { recursive: true }); + await writeFile(input.ledgerPath, `${JSON.stringify(ledger, null, 2)}\n`); + return ledger; +} + +export function createLiveCampaignFundingChain(input: { + funderPrivateKey: Hex; + contracts: CampaignContracts; +}): CampaignFundingChain { + const clients = createSeedClients(input.funderPrivateKey); + const token = { address: input.contracts.paymentToken, abi: PAYMENT_TOKEN_FUNDING_ABI }; + const wait = async (hash: Hex) => { + await clients.publicClient.waitForTransactionReceipt({ hash }); + return hash; + }; + return { + getNativeBalance: (address) => clients.publicClient.getBalance({ address }), + getTokenBalance: (address) => clients.publicClient.readContract({ ...token, functionName: 'balanceOf', args: [address] }), + async transferNative(to, amount) { + return wait(await clients.walletClient.sendTransaction({ to, value: amount })); + }, + async transferToken(to, amount) { + return wait(await clients.walletClient.writeContract({ + ...token, functionName: 'transfer', args: [to, amount], + chain: clients.walletClient.chain, account: clients.walletClient.account!, + })); + }, + async mintToken(to, amount) { + return wait(await clients.walletClient.writeContract({ + ...token, functionName: 'mintTo', args: [to, amount], + chain: clients.walletClient.chain, account: clients.walletClient.account!, + })); + }, + }; +} diff --git a/fake-data-generation/test/campaignProvisioning.test.ts b/fake-data-generation/test/campaignProvisioning.test.ts new file mode 100644 index 00000000..6a5f6cc6 --- /dev/null +++ b/fake-data-generation/test/campaignProvisioning.test.ts @@ -0,0 +1,110 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { parseEther, parseUnits, type Address, type Hex } from 'viem'; +import { privateKeyToAccount } from 'viem/accounts'; +import { LOCAL_HARDHAT_CHAIN_ID, type CampaignEnvironment } from '../campaignEnvironment.js'; +import type { CampaignPlan, PlannedAction } from '../campaignPlanner.js'; +import { computeCampaignFundingNeeds, provisionCampaignWallets, type CampaignFundingChain } from '../campaignProvisioning.js'; + +const generatedKey = '0x59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d' as const; +const generatedAddress = privateKeyToAccount(generatedKey).address; + +function action(partial: Partial & Pick): PlannedAction { + return { sequence: 1, actorUserId: 'user-001', dependsOn: [], ...partial }; +} + +const plan = { + campaignId: 'medium-realistic-v1', + users: [{ id: 'user-001', walletSlot: 'wallet-user-001' }], + actions: [ + action({ id: 'a1', type: 'set-belief' }), + action({ id: 'a2', type: 'fund-project', amount: 500 }), + action({ id: 'a3', type: 'deposit-note', amount: 1000 }), + ], +} as CampaignPlan; + +const wallet = { walletSlot: 'wallet-user-001', address: generatedAddress, privateKey: generatedKey, source: 'generated' as const }; + +function localEnv(): CampaignEnvironment { + return { + mode: 'local', rpcUrl: 'http://127.0.0.1:8545', expectedChainId: LOCAL_HARDHAT_CHAIN_ID, + contracts: {} as CampaignEnvironment['contracts'], + deployment: { strategy: 'existing-or-deploy' }, + provisioning: { walletSource: 'generated-or-hardhat', paymentTokenStrategy: 'transfer-or-mint' }, + }; +} + +test('funding needs include gas, note deposits, and project token buys', () => { + const [need] = computeCampaignFundingNeeds(plan, [wallet], 1_000_000_000n); + assert.equal(need.walletSlot, 'wallet-user-001'); + assert.ok(need.nativeWei > parseEther('0.05')); + assert.equal(need.paymentTokenUnits, parseUnits('0.01', 6)); + assert.ok(need.nativeWei >= parseEther('0.05') + 90_000n * 1_000_000_000n + 180_000n * 1_000_000_000n + 150_000n * 1_000_000_000n + parseEther('0.01')); +}); + +test('local provisioning mints when transfer fails and skips already-funded wallets', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-provision-')); + try { + const nativeTransfers: bigint[] = []; + const tokenMints: bigint[] = []; + const chain: CampaignFundingChain = { + getNativeBalance: async () => 0n, + getTokenBalance: async () => 0n, + transferNative: async (_to: Address, amount: bigint) => { nativeTransfers.push(amount); return '0x1' as Hex; }, + transferToken: async () => { throw new Error('insufficient allowance'); }, + mintToken: async (_to, amount) => { tokenMints.push(amount); return '0x2' as Hex; }, + }; + const ledger = await provisionCampaignWallets({ + environment: localEnv(), plan, wallets: [wallet], chain, ledgerPath: path.join(directory, 'ledger.json'), + }); + assert.equal(ledger.wallets[0].paymentTokenMethod, 'mint'); + assert.equal(nativeTransfers.length, 1); + assert.equal(tokenMints.length, 1); + const funded: CampaignFundingChain = { + getNativeBalance: async () => 10n ** 18n, + getTokenBalance: async () => 10n ** 18n, + transferNative: async () => { throw new Error('should skip native'); }, + transferToken: async () => { throw new Error('should skip token'); }, + }; + const skipped = await provisionCampaignWallets({ + environment: localEnv(), plan, wallets: [wallet], chain: funded, ledgerPath: path.join(directory, 'ledger.json'), + }); + assert.equal(skipped.wallets[0].nativeTransferred, false); + assert.equal(skipped.wallets[0].paymentTokenMethod, 'skipped'); + JSON.parse(await readFile(path.join(directory, 'ledger.json'), 'utf8')); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test('remote transfer-only provisioning refuses mint fallback and Hardhat wallets', async () => { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-provision-remote-')); + try { + const remote: CampaignEnvironment = { + mode: 'remote', rpcUrl: 'https://sepolia.base.org', expectedChainId: 84_532, + contracts: {} as CampaignEnvironment['contracts'], + deployment: { strategy: 'existing-only' }, + provisioning: { walletSource: 'generated-only', paymentTokenStrategy: 'transfer-only' }, + mutationConfirmed: true, + }; + await assert.rejects(provisionCampaignWallets({ + environment: remote, plan, wallets: [{ ...wallet, source: 'hardhat' }], + chain: { getNativeBalance: async () => 0n, getTokenBalance: async () => 0n, transferNative: async () => '0x1' as Hex, transferToken: async () => '0x1' as Hex }, + ledgerPath: path.join(directory, 'ledger.json'), + }), /refuses Hardhat wallets/); + await assert.rejects(provisionCampaignWallets({ + environment: remote, plan, wallets: [wallet], + chain: { + getNativeBalance: async () => 0n, getTokenBalance: async () => 0n, + transferNative: async () => '0x1' as Hex, + transferToken: async () => { throw new Error('funder empty'); }, + }, + ledgerPath: path.join(directory, 'ledger.json'), + }), /funder empty/); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); From 4673931a7b04ff6217e40a7f97c2ea3841c282b7 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 15:42:33 -0400 Subject: [PATCH 14/19] Fix campaign reconcile after the local 100-user run. SDK funding checks now use the adapter's 0.01 payment-token buy cost, and note lookups lowercase the bound contract address. The full local execute mined all 1932 actions; reconciliation is 1932/1932. --- CONTINUITY.md | 4 ++ fake-data-generation/README.md | 2 + .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignActionAdapter.ts | 4 +- fake-data-generation/campaignProvisioning.ts | 9 ++- .../campaignSdkDerivedChecks.ts | 10 ++-- .../medium-realistic-v1-local-run.md | 34 ++++++++++++ fake-data-generation/paymentTokenUnits.ts | 7 +++ .../test/campaignSdkDerivedChecks.test.ts | 55 ++++++++++++++++++- 9 files changed, 112 insertions(+), 15 deletions(-) create mode 100644 fake-data-generation/campaigns/medium-realistic-v1-local-run.md diff --git a/CONTINUITY.md b/CONTINUITY.md index f4180b1f..1fcfd1f2 100644 --- a/CONTINUITY.md +++ b/CONTINUITY.md @@ -2155,3 +2155,7 @@ Item 6 of the medium-realistic campaign: indexer adapter paginates with `fetchEv ## 2026-09-11 — Local campaign wallet provisioning Item 7 of the medium-realistic campaign: `campaignProvisioning.ts` funds generated wallets before `gen:campaign:execute`. Local mode transfers ETH from the Hardhat funder and payment tokens (mint fallback). Remote stays transfer-only and refuses Hardhat keys. `--skip-provision` exists for already-funded reruns. Live smoke on the local stack wrote `execution/funding-ledger.json` for all 100 slots. Full 100-user execute/reconcile/UI inspection is still open. + +## 2026-09-11 — Full local 100-user campaign execute + reconcile + +Ran `gen:campaign:execute --mode local --concurrency 4 --skip-provision`: 1932 mined, 0 failed, ~196s. First reconcile mismatched all fund-project and note SDK folds. Fixes: derived funding checks now use `campaignFundProjectCost()` (0.01 token per buy, matching the adapter) instead of planner persona amounts; note lookups lowercase the bound contract address. Re-reconcile: 1932/1932 verified, indexer lag 0. CauseStarter UI on this host still talks to Base Sepolia (`chainId=84532`), so browser inspection of campaign boards is blocked until the UI points at 31337. Notes: `fake-data-generation/campaigns/medium-realistic-v1-local-run.md`. Plan item 7 remains open for that UI pass. diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index 693ab934..e5aa5e0a 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -17,6 +17,8 @@ Campaign execution uses the explicit environment boundary in `campaignEnvironmen The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) is the contract-binding layer: it maps each planned action type onto SDK writes, classifies RPC vs revert failures, and updates public runtime bindings (no private keys). `npm run gen:campaign:execute` runs that adapter locally after environment preflight and funds generated wallets (ETH plus payment tokens, minting locally when transfer fails). Remote mode still requires `--confirm-remote-mutation` and stays transfer-only. Pass `--skip-provision` only when wallets are already funded. Generated wallet secrets stay under `output/campaigns/secrets/` (gitignored via `output/`). The funding ledger is `execution/funding-ledger.json`. +Each local `fund-project` write currently spends `CAMPAIGN_FUND_PROJECT_TOKEN` (`0.01` of the 6-decimal payment token), not the planner’s persona-sized `amount`. Reconciliation’s SDK funding check uses that same unit cost. Note folds are looked up as `:`. + `campaignReconciler.ts` provides the matching transaction-to-indexer boundary. `campaignIndexerAdapter.ts` binds every campaign action type to its real raw Ponder event(s), reads chain/indexer heads, and delegates derived truth to `campaignSdkDerivedChecks.ts`. Those binding-aware SDK checks verify final folded beliefs, implications, cause refs, projects and cumulative funding, alignments, and note ownership; repeated writes compare with the campaign's final intended state. The reusable reconciler detects omissions, duplicates, and derived-state mismatches, polls through a bounded settling window, records chain-head lag and indexing latency, and atomically writes machine-readable JSON plus a concise operator summary. `campaignRuntimeBindings.ts` defines the public, execution-produced handoff from planned IDs to wallet addresses, statement/roster CIDs, project contracts, and on-chain note IDs; it validates plan identity and coverage and persists atomically without wallet secrets. `npm run gen:campaign:reconcile -- --probe` talks to the live local chain and Ponder `/status` heads. The same command without `--probe` loads planned actions, execution state, and runtime bindings, then writes the reconciliation JSON/summary through the real indexer adapter and SDK folds. Raw lookups paginate with `fetchEventsComplete` and treat ordinary vs retroactive funding events as one proving write. A live adapter test runs when `http://localhost:42069` is reachable. Full 100-user local execution provisions wallets beyond the funded Hardhat accounts before submitting writes; a dry adapter test covers binding updates without a live chain. ## Overview diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index f4d3532e..e69aa2ea 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -183,7 +183,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. 6. **[x] Bind execution to contracts and deep reconciliation.** [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) turns planned actions into contract writes, records mined hashes, and updates the public runtime-binding artifact. `npm run gen:campaign:execute` is local-first, refuses remote mutation without `--confirm-remote-mutation`, and keeps wallet secrets outside the campaign directory. Reconciliation already maps those hashes to Ponder events and SDK folds. Local execution now funds generated wallets (ETH plus payment tokens, with mint fallback) and writes `execution/funding-ledger.json` before submitting writes. -7. **[ ] Run and inspect the full 100-user campaign locally.** Exercise the deep local stack, fix harness defects, record runtime/resource baselines, and inspect representative pages in the browser. Adjust the workload only through reviewed manifest changes. +7. **[ ] Run and inspect the full 100-user campaign locally.** Local execute mined all 1932 actions in ~196s (concurrency 4); after harness fixes, reconciliation is 1932/1932 with lag 0. See [`campaigns/medium-realistic-v1-local-run.md`](./campaigns/medium-realistic-v1-local-run.md). Remaining: inspect representative CauseStarter pages against a UI pointed at local chain 31337 (current Vite/IPFS bundles were on Base Sepolia). 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. 10. **[ ] Run staged 25-user then 100-user campaigns.** Re-run health/preflight and reconcile after each phase. Do not automatically advance after a failed or materially surprising phase. diff --git a/fake-data-generation/campaignActionAdapter.ts b/fake-data-generation/campaignActionAdapter.ts index 14bbec89..ebf6e1fa 100644 --- a/fake-data-generation/campaignActionAdapter.ts +++ b/fake-data-generation/campaignActionAdapter.ts @@ -26,7 +26,7 @@ import type { CampaignRuntimeBindings } from './campaignRuntimeBindings.js'; import { writeRuntimeBindings } from './campaignRuntimeBindings.js'; import { buildSeedRosterDocument } from './seedCauseRoster.js'; import { createSeedClients } from './seedRpc.js'; -import { getPaymentTokenDecimals } from './paymentTokenUnits.js'; +import { campaignFundProjectCost, getPaymentTokenDecimals } from './paymentTokenUnits.js'; const GAS_UNITS: Record = { 'publish-statement': 180_000n, 'create-cause': 120_000n, 'set-belief': 90_000n, @@ -188,7 +188,7 @@ export function createLiveCampaignActionWriter(input: { token = folded.erc1155Address as Address; projectTokens.set(action.projectId!, token); } - return { hash: await buyProjectTokens(clients, { address: assurance, abi: AssuranceContractAbi }, { buyer: clients.account, tokenAddress: token, tokenIds: [3n], tokenCounts: [1n], totalCost: parseUnits('0.01', decimals) }) }; + return { hash: await buyProjectTokens(clients, { address: assurance, abi: AssuranceContractAbi }, { buyer: clients.account, tokenAddress: token, tokenIds: [3n], tokenCounts: [1n], totalCost: campaignFundProjectCost() }) }; }, async 'deposit-note'(action, clients) { const { hash, noteId } = await depositETH(clients, notesContract, { amount: noteAmount(action) }); diff --git a/fake-data-generation/campaignProvisioning.ts b/fake-data-generation/campaignProvisioning.ts index 981f896a..63a9c34f 100644 --- a/fake-data-generation/campaignProvisioning.ts +++ b/fake-data-generation/campaignProvisioning.ts @@ -1,9 +1,9 @@ import { mkdir, writeFile } from 'node:fs/promises'; import path from 'node:path'; -import { parseEther, parseUnits, type Address, type Hex } from 'viem'; +import { parseEther, type Address, type Hex } from 'viem'; import type { CampaignContracts, CampaignEnvironment, CampaignWalletBinding } from './campaignEnvironment.js'; import type { CampaignPlan, PlannedAction } from './campaignPlanner.js'; -import { getPaymentTokenDecimals } from './paymentTokenUnits.js'; +import { campaignFundProjectCost } from './paymentTokenUnits.js'; import { createSeedClients } from './seedRpc.js'; const GAS_UNITS: Record = { @@ -14,7 +14,7 @@ const GAS_UNITS: Record = { const DEFAULT_GAS_PRICE = 1_000_000_000n; const NATIVE_BUFFER_WEI = parseEther('0.05'); -const FUND_PROJECT_TOKEN = '0.01'; + export const PAYMENT_TOKEN_FUNDING_ABI = [ { @@ -85,8 +85,7 @@ export function computeCampaignFundingNeeds(plan: CampaignPlan, wallets: readonl for (const wallet of wallets) { bySlot.set(wallet.walletSlot, { walletSlot: wallet.walletSlot, address: wallet.address, nativeWei: NATIVE_BUFFER_WEI, paymentTokenUnits: 0n }); } - const decimals = getPaymentTokenDecimals(); - const fundCost = parseUnits(FUND_PROJECT_TOKEN, decimals); + const fundCost = campaignFundProjectCost(); for (const action of plan.actions) { const user = action.actorUserId ? users.get(action.actorUserId) : undefined; if (!user) continue; diff --git a/fake-data-generation/campaignSdkDerivedChecks.ts b/fake-data-generation/campaignSdkDerivedChecks.ts index b0561990..6054d278 100644 --- a/fake-data-generation/campaignSdkDerivedChecks.ts +++ b/fake-data-generation/campaignSdkDerivedChecks.ts @@ -10,6 +10,7 @@ import type { CampaignDerivedCheckProvider } from './campaignIndexerAdapter.js'; import type { CampaignPlan, PlannedAction } from './campaignPlanner.js'; import { validateRuntimeBindings, type CampaignRuntimeBindings } from './campaignRuntimeBindings.js'; import type { DerivedCheck } from './campaignReconciler.js'; +import { campaignFundProjectCost } from './paymentTokenUnits.js'; /** Injectable SDK query surface, primarily to make the expected-state logic testable. */ export interface CampaignSdkQueries { @@ -58,11 +59,12 @@ export function createCampaignSdkDerivedCheckProvider(input: { const queries = input.queries ?? realSdkQueries(input.machinery); const latestBelief = new Map(); const latestNoteAction = new Map(); - const fundingByProject = new Map(); + const fundingByProject = new Map(); + const fundCost = campaignFundProjectCost(); for (const action of plan.actions) { if (action.type === 'set-belief') latestBelief.set(`${action.actorUserId}/${action.statementId}`, action); if (action.noteId) latestNoteAction.set(action.noteId, action); - if (action.type === 'fund-project') fundingByProject.set(action.projectId!, (fundingByProject.get(action.projectId!) ?? 0) + action.amount!); + if (action.type === 'fund-project') fundingByProject.set(action.projectId!, (fundingByProject.get(action.projectId!) ?? 0n) + fundCost); } const user = (id: string | null | undefined): Address => bindings.users[id!]; @@ -97,13 +99,13 @@ export function createCampaignSdkDerivedCheckProvider(input: { return [check('SDK active project alignment', true, await queries.hasAlignment(user(action.actorUserId), project(action.projectId), statement(action.statementId)))]; case 'fund-project': { const folded = await queries.getProject(project(action.projectId)); - return [check('SDK final project funding', String(fundingByProject.get(action.projectId!) ?? 0), folded?.totalReceived ?? null)]; + return [check('SDK final project funding', (fundingByProject.get(action.projectId!) ?? 0n).toString(), folded?.totalReceived ?? null)]; } case 'deposit-note': case 'delegate-note': case 'revoke-delegation': { const binding = bindings.notes[action.noteId!]; - const folded = await queries.getNote(`${binding.contractAddress}:${binding.noteId}`); + const folded = await queries.getNote(`${binding.contractAddress.toLowerCase()}:${binding.noteId}`); const final = latestNoteAction.get(action.noteId!)!; const expectedOwner = final.type === 'delegate-note' ? user(final.delegateUserId) : user(final.actorUserId); const rootOwner = user(plan.actions.find((item) => item.type === 'deposit-note' && item.noteId === action.noteId)!.actorUserId); diff --git a/fake-data-generation/campaigns/medium-realistic-v1-local-run.md b/fake-data-generation/campaigns/medium-realistic-v1-local-run.md new file mode 100644 index 00000000..ea1e4a63 --- /dev/null +++ b/fake-data-generation/campaigns/medium-realistic-v1-local-run.md @@ -0,0 +1,34 @@ +# Medium realistic v1 — local 100-user run notes + +Operator notes from the first full local execute/reconcile (2026-09-11). Artifacts under `output/campaigns/medium-realistic-v1/` are gitignored. + +## Execution + +- Command: `npm run gen:campaign:execute -- --mode local --concurrency 4 --skip-provision` from `fake-data-generation/` (wallets already funded). +- Resume: previous state had 3 mined / 1929 planned; runner finished the rest. +- Result: **1932 mined, 0 failed**, wall clock **~196s** at concurrency 4, pacing 0. +- Chain / indexer after settle: head **3044**, lag **0**. +- Gas used (sum of receipts): **252,914,126**. Native cost at recorded gas price: **~0.253 ETH**. +- Funding: local Hardhat keys for the first slots plus generated wallets; payment-token buys used **0.01** of the 6-decimal token per `fund-project` write (885 buys), not the planner’s persona-sized integer amounts. + +## Reconciliation + +First pass reported `derived-mismatch` for every `fund-project` and note action (777/1932 verified). Causes: + +1. SDK funding checks summed planner `amount` values; the adapter always buys `CAMPAIGN_FUND_PROJECT_TOKEN` (`0.01`). Checks now use that cost times write count. +2. Note folds key notes as `:`; lookups used the checksummed address and missed. + +After those harness fixes, `npm run gen:campaign:reconcile` reported **1932/1932 verified**, no missing/duplicate indexed events. + +## UI inspection + +CauseStarter has no global cause directory; campaign boards are reachable only by organizer URL: + +`/cause//` from `execution/runtime-bindings.json` (example: schools-common-ground owner `0x29Aad1ae4EC538790a3231c62d84d6840685D613`, ref `campaign-medium-realistic-v1-schools-common-ground`). + +On this machine the Vite app (`:5174`) and the published IPFS bundle (`http://causestarter.localhost:8088/#/`) both requested `chainId=84532` at `https://commonality-indexer.onrender.com`. Cause pages showed “Failed to fetch” (CORS). That is the root `.env` / `causestarter/.env` pointing at Base Sepolia, not a campaign indexing omission. Re-inspect after pointing the UI at chain 31337 and `http://localhost:42069`. + +## Still open for plan item 7 + +- Browser pass against a local-chain UI (cause board, a funded project, a statement, a note/delegate path). +- Optional: make `fund-project` writes use planned persona amounts (would require a wipe/re-execute so totals stay reconcilable). diff --git a/fake-data-generation/paymentTokenUnits.ts b/fake-data-generation/paymentTokenUnits.ts index 67eb9cba..a9215b1d 100644 --- a/fake-data-generation/paymentTokenUnits.ts +++ b/fake-data-generation/paymentTokenUnits.ts @@ -17,3 +17,10 @@ export function getPaymentTokenDecimals(): number { export function parsePaymentTokenUnits(value: string): bigint { return parseUnits(value, getPaymentTokenDecimals()); } + +/** Each campaign fund-project write currently buys this payment-token amount. */ +export const CAMPAIGN_FUND_PROJECT_TOKEN = '0.01'; + +export function campaignFundProjectCost(): bigint { + return parsePaymentTokenUnits(CAMPAIGN_FUND_PROJECT_TOKEN); +} diff --git a/fake-data-generation/test/campaignSdkDerivedChecks.test.ts b/fake-data-generation/test/campaignSdkDerivedChecks.test.ts index 4d06f658..6615e39f 100644 --- a/fake-data-generation/test/campaignSdkDerivedChecks.test.ts +++ b/fake-data-generation/test/campaignSdkDerivedChecks.test.ts @@ -40,12 +40,13 @@ test('SDK provider checks final folded state through runtime bindings', async () getProject: async (projectAddress) => { calls.push(projectAddress); const projectId = Object.entries(bindings.projects).find(([, value]) => value === projectAddress)![0]; - const total = plan.actions.filter((item) => item.type === 'fund-project' && item.projectId === projectId).reduce((sum, item) => sum + item.amount!, 0); - return { id: projectAddress, erc1155Address: address(999), marketplaceAddress: null, recipient: address(998), fundingCurrency: { chainId: 31337, tokenAddress: address(997), symbol: 'TEST', decimals: 0 }, threshold: '1', deadline: '1', totalReceived: String(total), conditionAddress: null }; + const { campaignFundProjectCost } = await import('../paymentTokenUnits.js'); + const count = plan.actions.filter((item) => item.type === 'fund-project' && item.projectId === projectId).length; + return { id: projectAddress, erc1155Address: address(999), marketplaceAddress: null, recipient: address(998), fundingCurrency: { chainId: 31337, tokenAddress: address(997), symbol: 'TEST', decimals: 0 }, threshold: '1', deadline: '1', totalReceived: (campaignFundProjectCost() * BigInt(count)).toString(), conditionAddress: null }; }, hasAlignment: async () => true, getNote: async (scopedId) => { - const bindingEntry = Object.entries(bindings.notes).find(([, value]) => `${value.contractAddress}:${value.noteId}` === scopedId)!; + const bindingEntry = Object.entries(bindings.notes).find(([, value]) => `${value.contractAddress.toLowerCase()}:${value.noteId}` === scopedId)!; const deposit = plan.actions.find((item) => item.type === 'deposit-note' && item.noteId === bindingEntry[0])!; const final = plan.actions.filter((item) => item.noteId === bindingEntry[0]).at(-1)!; const owner = final.type === 'delegate-note' ? bindings.users[final.delegateUserId!] : bindings.users[final.actorUserId!]; @@ -61,6 +62,54 @@ test('SDK provider checks final folded state through runtime bindings', async () assert.ok(calls.length > 0); }); +test('SDK note lookup lowercases the bound contract address', async () => { + const manifest = JSON.parse(await readFile(new URL('../campaigns/medium-realistic-v1.json', import.meta.url), 'utf8')) as CampaignManifestV1; + const plan = await buildCampaignPlan(manifest); + const deposit = plan.actions.find((item) => item.type === 'deposit-note')!; + const mixed = '0x8A791620dd6260079BF849Dc5567aDC3F2FdC318' as Address; + const bindings: CampaignRuntimeBindings = { + version: CAMPAIGN_RUNTIME_BINDINGS_VERSION, + campaignId: plan.campaignId, + manifestFingerprint: plan.manifestFingerprint, + updatedAt: new Date(0).toISOString(), + users: Object.fromEntries(plan.users.map((item, index) => [item.id, address(index + 1)])), + statements: Object.fromEntries(plan.statements.map((item) => [item.id, fakeIpfsCidV1(item.id)])), + causes: Object.fromEntries([...new Set(plan.statements.map((item) => item.causeId))].map((id, index) => [id, { owner: address(index + 201), refName: `cause-${id}`, rosterCid: fakeIpfsCidV1(`roster-${id}`) }])), + projects: Object.fromEntries(plan.projects.map((item, index) => [item.id, address(index + 301)])), + notes: Object.fromEntries([...new Set(plan.actions.flatMap((item) => item.noteId ? [item.noteId] : []))].map((id, index) => [id, { contractAddress: mixed, noteId: String(index + 1) }])), + }; + const seen: string[] = []; + const queries = { + getUserBelief: async () => 1, + hasImplication: async () => true, + getRefsByName: async () => [], + getProject: async () => null, + hasAlignment: async () => true, + getNote: async (id) => { + seen.push(id); + return { + id: bindings.notes[deposit.noteId!].noteId, + contractAddress: mixed, + chainHash: '0x', + amount: '1', + token: address(996), + tokenType: 0, + tokenId: '0', + owner: bindings.users[deposit.actorUserId!], + rootOwner: bindings.users[deposit.actorUserId!], + active: true, + createdAt: '0', + createdAtBlock: '1', + updatedAt: '0', + }; + }, + } satisfies CampaignSdkQueries; + const provider = createCampaignSdkDerivedCheckProvider({ machinery: { ipfsConfig: {}, twitterApiConfig: {}, testConfig: {} }, plan, bindings, queries }); + const checks = await provider.getDerivedChecks(deposit); + assert.equal(seen[0], `${mixed.toLowerCase()}:${bindings.notes[deposit.noteId!].noteId}`); + assert.equal(checks[0].actual, true); +}); + test('SDK provider exposes a derived mismatch instead of hiding it', async () => { const manifest = JSON.parse(await readFile(new URL('../campaigns/medium-realistic-v1.json', import.meta.url), 'utf8')) as CampaignManifestV1; const plan = await buildCampaignPlan(manifest); From a25820cf4f073e0ca02c3010b84fc42f959af490 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 15:55:24 -0400 Subject: [PATCH 15/19] File a TODO for unmixing local and testnet generated env. A fresh LLM should make localhost and Base Sepolia complete separate profiles so --start and testnet publish stop sharing one corrupted .env tree. --- TODO.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/TODO.md b/TODO.md index f3a4bfed..702afd24 100644 --- a/TODO.md +++ b/TODO.md @@ -20,6 +20,46 @@ Getting **testnet to a two-person shared lab** is also a standing plan, not a pi ---- +- Stop mixing localhost and Base Sepolia in the generated env files so a laptop + can run a complete local stack **and** publish a complete testnet UI without + one clobbering the other. This is not “one SPA talks to both chains.” Live + Render testnet already has its own env; the bug is this checkout’s generated + `.env` / `ui/.env` / `causestarter/.env`. + + **What went wrong (2026-09-11, medium-realistic local campaign):** those files + were last written by `scripts/setup-env.sh base-sepolia` (`CHAIN_ID=84532`, + `VITE_EVENT_CACHE_URL=https://commonality-indexer.onrender.com`) but carried + Hardhat contract addresses and a localhost IPFS gateway. `scripts/services.sh + --start` loads `deployments/localhost.env` then lets the live env files win + (`map_causestarter_contract_env` keeps an already-set `VITE_CHAIN_ID`). + CauseStarter Docker writes `config.json` from that mix + (`causestarter/docker-entrypoint.d/40-causestarter-config.sh`). The local UI + gateway (`scripts/local-ui-gateway.mjs`) only proxies `/api` when the bundle + **omits** `VITE_EVENT_CACHE_URL`; a baked Render URL bypasses it. Vite on + `:5174` and `http://causestarter.localhost:8088/` then requested chain 84532 + and showed “Failed to fetch” (CORS) while the campaign had populated chain + 31337. Same class of bug as + [chain-scoped-trust-config.md](docs/dev/chain-scoped-trust-config.md). + + **Do this:** + 1. Keep localhost vs Base Sepolia as two complete generated profiles. Do not + overlay a later file from a different chain onto an earlier one. + 2. `./scripts/services.sh --start` must **activate localhost**: chain 31337, + `http://127.0.0.1:8545`, local indexer (or omit `VITE_EVENT_CACHE_URL` so + `:8088` can proxy). Refuse to start if chain id, RPC, indexer URL, and + contract bytecode disagree. + 3. Testnet publish (`setup-env.sh base-sepolia` plus `deploy-ui` / + `deploy-causestarter`) must not leave the laptop stuck on the testnet + profile — generate into a per-network dir or temp tree, or restore + localhost after the publish. + 4. Add a cheap consistency check (start and/or verifier) so a mixed profile + fails loudly instead of rendering empty. + + Context: [medium-realistic-v1-local-run.md](fake-data-generation/campaigns/medium-realistic-v1-local-run.md), + `scripts/setup-env.sh`, `scripts/services.sh` (`map_causestarter_contract_env`), + `workflow/local-development.md`, `workflow/deployment.md`. Do not invent a + runtime chain switcher in the SPA. + - Align `foldReimbursements` donation rounding with the contract’s per-share accumulator (`accumulatedReimbursementPerClaimShare` / `mulDiv`). The fold currently splits each donation with per-holder `claim * amount / outstanding` From fafc61e304544b2537eb78e40175a97001d479f8 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 16:06:48 -0400 Subject: [PATCH 16/19] Keep localhost and testnet generated env profiles from mixing. Local start now regenerates a complete localhost profile and refuses mixed chain/RPC/indexer settings. Testnet UI publishes snapshot into .generated-env and restore localhost on exit so a laptop can run both without clobbering the live env files. --- .gitignore | 1 + TODO.md | 40 --------------------- scripts/check-local-config-sync.mjs | 35 +++++++++++++++++++ scripts/deploy-causestarter.sh | 5 ++- scripts/deploy-testnet-uber-ui.sh | 5 +++ scripts/deploy-ui.sh | 5 +++ scripts/services.sh | 3 ++ scripts/setup-env.sh | 54 +++++++++++++++++++++-------- workflow/deployment.md | 5 +++ workflow/local-development.md | 5 +++ 10 files changed, 103 insertions(+), 55 deletions(-) diff --git a/.gitignore b/.gitignore index 70ed0dce..6fab4c0d 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,7 @@ node_modules .env.secrets .env.render .env.cloudflare +.generated-env/ services/implication-attester/.env ui/.env integration-tests/.env.local diff --git a/TODO.md b/TODO.md index 702afd24..f3a4bfed 100644 --- a/TODO.md +++ b/TODO.md @@ -20,46 +20,6 @@ Getting **testnet to a two-person shared lab** is also a standing plan, not a pi ---- -- Stop mixing localhost and Base Sepolia in the generated env files so a laptop - can run a complete local stack **and** publish a complete testnet UI without - one clobbering the other. This is not “one SPA talks to both chains.” Live - Render testnet already has its own env; the bug is this checkout’s generated - `.env` / `ui/.env` / `causestarter/.env`. - - **What went wrong (2026-09-11, medium-realistic local campaign):** those files - were last written by `scripts/setup-env.sh base-sepolia` (`CHAIN_ID=84532`, - `VITE_EVENT_CACHE_URL=https://commonality-indexer.onrender.com`) but carried - Hardhat contract addresses and a localhost IPFS gateway. `scripts/services.sh - --start` loads `deployments/localhost.env` then lets the live env files win - (`map_causestarter_contract_env` keeps an already-set `VITE_CHAIN_ID`). - CauseStarter Docker writes `config.json` from that mix - (`causestarter/docker-entrypoint.d/40-causestarter-config.sh`). The local UI - gateway (`scripts/local-ui-gateway.mjs`) only proxies `/api` when the bundle - **omits** `VITE_EVENT_CACHE_URL`; a baked Render URL bypasses it. Vite on - `:5174` and `http://causestarter.localhost:8088/` then requested chain 84532 - and showed “Failed to fetch” (CORS) while the campaign had populated chain - 31337. Same class of bug as - [chain-scoped-trust-config.md](docs/dev/chain-scoped-trust-config.md). - - **Do this:** - 1. Keep localhost vs Base Sepolia as two complete generated profiles. Do not - overlay a later file from a different chain onto an earlier one. - 2. `./scripts/services.sh --start` must **activate localhost**: chain 31337, - `http://127.0.0.1:8545`, local indexer (or omit `VITE_EVENT_CACHE_URL` so - `:8088` can proxy). Refuse to start if chain id, RPC, indexer URL, and - contract bytecode disagree. - 3. Testnet publish (`setup-env.sh base-sepolia` plus `deploy-ui` / - `deploy-causestarter`) must not leave the laptop stuck on the testnet - profile — generate into a per-network dir or temp tree, or restore - localhost after the publish. - 4. Add a cheap consistency check (start and/or verifier) so a mixed profile - fails loudly instead of rendering empty. - - Context: [medium-realistic-v1-local-run.md](fake-data-generation/campaigns/medium-realistic-v1-local-run.md), - `scripts/setup-env.sh`, `scripts/services.sh` (`map_causestarter_contract_env`), - `workflow/local-development.md`, `workflow/deployment.md`. Do not invent a - runtime chain switcher in the SPA. - - Align `foldReimbursements` donation rounding with the contract’s per-share accumulator (`accumulatedReimbursementPerClaimShare` / `mulDiv`). The fold currently splits each donation with per-holder `claim * amount / outstanding` diff --git a/scripts/check-local-config-sync.mjs b/scripts/check-local-config-sync.mjs index 0681bfb3..b0f0feb3 100755 --- a/scripts/check-local-config-sync.mjs +++ b/scripts/check-local-config-sync.mjs @@ -195,6 +195,27 @@ function checkViteMirror(rootEnv, viteEnv, label) { } } +function checkLocalIdentity(env, label, { vite = false } = {}) { + const chainId = vite ? env.VITE_CHAIN_ID : env.CHAIN_ID; + const rpcUrl = vite ? env.VITE_ETH_RPC_URL : env.ETHEREUM_RPC_URL; + const eventCacheUrl = vite ? env.VITE_EVENT_CACHE_URL : env.EVENT_CACHE_URL; + + if (chainId !== '31337') errors.push(`${label}: expected local chain id 31337, got ${chainId || '(missing)'}`); + if (env.COMMONALITY_ENVIRONMENT !== 'local') { + errors.push(`${label}: expected COMMONALITY_ENVIRONMENT=local, got ${env.COMMONALITY_ENVIRONMENT || '(missing)'}`); + } + if (!/^http:\/\/(127\.0\.0\.1|localhost):8545\/?$/.test(rpcUrl || '')) { + errors.push(`${label}: expected localhost RPC on port 8545, got ${rpcUrl || '(missing)'}`); + } + if (vite) { + if (eventCacheUrl && !/^http:\/\/(127\.0\.0\.1|localhost):42069\/?$/.test(eventCacheUrl)) { + errors.push(`${label}: VITE_EVENT_CACHE_URL points off-laptop (${eventCacheUrl}); leave it empty for the local proxy`); + } + } else if (!/^http:\/\/(127\.0\.0\.1|localhost):42069\/?$/.test(eventCacheUrl || '')) { + errors.push(`${label}: expected local EVENT_CACHE_URL on port 42069, got ${eventCacheUrl || '(missing)'}`); + } +} + function checkRuntimeAgainstDeploy(name, config, deployEnv) { for (const [rootKey, viteKey] of Object.entries(ROOT_TO_VITE)) { if (!REQUIRED_ROOT_KEYS.includes(rootKey) && rootKey !== 'NUDGE_PUBLICATIONS_CONTRACT_ADDRESS') { @@ -226,6 +247,17 @@ async function checkChain(rpcUrl, deployEnv) { } notes.push(`RPC ${rpcUrl} ok (block ${Number.parseInt(blockNumber, 16)})`); + try { + const actualChainId = Number.parseInt(await rpc(rpcUrl, 'eth_chainId'), 16); + if (actualChainId !== 31337) { + errors.push(`RPC ${rpcUrl} reports chain ${actualChainId}, expected local chain 31337`); + return; + } + } catch (err) { + errors.push(`Could not read chain id from ${rpcUrl}: ${err.message}`); + return; + } + for (const key of REQUIRED_ROOT_KEYS) { const address = deployEnv[key]; if (!address || !isAddress(address)) continue; @@ -306,6 +338,7 @@ async function main() { if (rootEnv) { log(` loaded ${path.relative(ROOT, rootEnvPath)}`); + checkLocalIdentity(rootEnv, '.env'); // Root .env should mirror deploy addresses when present. if (localhostEnv) { for (const key of REQUIRED_ROOT_KEYS) { @@ -325,12 +358,14 @@ async function main() { if (uiEnv) { log(` loaded ${path.relative(ROOT, uiEnvPath)}`); + checkLocalIdentity(uiEnv, 'ui/.env', { vite: true }); checkViteMirror(deployEnv, uiEnv, 'ui/.env'); } else { warnings.push('ui/.env missing — UI IPFS publisher may bake empty contract addresses'); } if (causestarterEnv) { + checkLocalIdentity(causestarterEnv, 'causestarter/.env', { vite: true }); // Optional package env; only check drift if it sets addresses. const hasAny = Object.values(ROOT_TO_VITE).some((k) => causestarterEnv[k]); if (hasAny) { diff --git a/scripts/deploy-causestarter.sh b/scripts/deploy-causestarter.sh index 3b4a676a..3bb35ea3 100755 --- a/scripts/deploy-causestarter.sh +++ b/scripts/deploy-causestarter.sh @@ -122,7 +122,10 @@ if ! docker info >/dev/null 2>&1; then exit 1 fi -# localhost.env matches hardhat-deploy --network localhost; live .env files win. +"$ROOT/scripts/setup-env.sh" localhost +"$ROOT/scripts/check-local-config-sync.sh" --env-only + +# All live files now belong to the same localhost profile. load_env_file "$ROOT/deployments/localhost.env" load_env_file "$ROOT/.env" load_env_file "$ROOT/ui/.env" diff --git a/scripts/deploy-testnet-uber-ui.sh b/scripts/deploy-testnet-uber-ui.sh index e5f904d2..6088a675 100755 --- a/scripts/deploy-testnet-uber-ui.sh +++ b/scripts/deploy-testnet-uber-ui.sh @@ -126,6 +126,11 @@ json_app_entry() { echo "Setting up environment for $NETWORK..." "$ROOT/scripts/setup-env.sh" "$NETWORK" +restore_local_profile() { + echo "Restoring the localhost environment profile..." + "$ROOT/scripts/setup-env.sh" localhost +} +trap restore_local_profile EXIT EVENT_CACHE_URL=$(grep -E '^VITE_EVENT_CACHE_URL=' "$ROOT/ui/.env" | tail -1 | cut -d= -f2-) if [ -z "$EVENT_CACHE_URL" ]; then diff --git a/scripts/deploy-ui.sh b/scripts/deploy-ui.sh index ad54ea9a..c90ddb64 100755 --- a/scripts/deploy-ui.sh +++ b/scripts/deploy-ui.sh @@ -64,6 +64,11 @@ fi # --- Set up environment for the target network --- echo "Setting up environment for $NETWORK..." "$ROOT/scripts/setup-env.sh" "$NETWORK" +restore_local_profile() { + echo "Restoring the localhost environment profile..." + "$ROOT/scripts/setup-env.sh" localhost +} +trap restore_local_profile EXIT # The IPFS bundle cannot rely on a dev-server proxy. The event cache URL is # baked into the Vite build and must point at the deployed indexer. diff --git a/scripts/services.sh b/scripts/services.sh index a22deb22..2f5070a3 100755 --- a/scripts/services.sh +++ b/scripts/services.sh @@ -326,6 +326,9 @@ start_services() { local -a services_to_build=() timing_begin + echo "[$(date +%T)] Activating the complete localhost environment profile..." + "$SCRIPT_DIR/setup-env.sh" localhost + "$SCRIPT_DIR/check-local-config-sync.sh" --env-only "$SCRIPT_DIR/check-prerequisites.sh" check_existing_containers clear_stale_ponder_for_fresh_chain diff --git a/scripts/setup-env.sh b/scripts/setup-env.sh index 70c6a872..a446c354 100755 --- a/scripts/setup-env.sh +++ b/scripts/setup-env.sh @@ -54,11 +54,10 @@ load_env_file() { done <"$file" } -load_env_file "$DEPLOYMENT_FILE" -# Local: operator-addresses is the identity source (Hardhat keys). -# Testnet/mainnet: deployments/.env is the chain of record for public -# attester/nudger/assist URLs. operator-addresses.env is often a leftover local -# generate-wallets.mjs dump and must not clobber Sepolia identities. +# Secrets provide credentials and optional service settings. Public values from +# the selected deployment are loaded last so stale cross-chain values cannot +# override the selected chain's contract addresses. +load_env_file "$SECRETS_FILE" load_env_file_fill_missing() { local file="$1" [ -f "$file" ] || return 0 @@ -76,7 +75,7 @@ if [ "$NETWORK" = "localhost" ]; then else load_env_file_fill_missing "$WALLETS_FILE" fi -load_env_file "$SECRETS_FILE" +load_env_file "$DEPLOYMENT_FILE" DOMAIN_SLUGS=(commonality lazygiving alignment tally content-funding civility common-sense-majority conceptspace causestarter) DOMAIN_URL_VARS=(VITE_COMMONALITY_URL VITE_LAZYGIVING_URL VITE_ALIGNMENT_URL VITE_TALLY_URL VITE_CONTENT_FUNDING_URL VITE_CIVILITY_URL VITE_COMMON_SENSE_MAJORITY_URL VITE_CONCEPTSPACE_URL VITE_CAUSESTARTER_URL) @@ -142,17 +141,22 @@ case "$NETWORK" in localhost) VARS[COMMONALITY_ENVIRONMENT]="local" VARS[CHAIN_ID]="31337" - VARS[ETHEREUM_RPC_URL]="${VARS[ETHEREUM_RPC_URL]:-http://localhost:8545}" - VARS[IPFS_API]="${VARS[IPFS_API]:-http://localhost:5001}" - VARS[IPFS_GATEWAY]="${VARS[IPFS_GATEWAY]:-http://localhost:8080/ipfs}" - VARS[EVENT_CACHE_URL]="${VARS[EVENT_CACHE_URL]:-http://localhost:42069}" - VARS[PLATFORM_API_URL]="${VARS[PLATFORM_API_URL]:-http://localhost:3001}" + VARS[ETHEREUM_RPC_URL]="http://127.0.0.1:8545" + VARS[IPFS_API]="http://localhost:5001" + VARS[IPFS_GATEWAY]="http://localhost:8080/ipfs" + VARS[EVENT_CACHE_URL]="http://localhost:42069" + VARS[UI_EVENT_CACHE_URL]="" + VARS[PLATFORM_API_URL]="http://localhost:3001" + VARS[VITE_DEFAULT_TRUSTED_ATTESTERS]="" + VARS[VITE_DEFAULT_TRUSTED_CONTENT_ATTESTERS]="" + VARS[VITE_DEFAULT_TRUSTED_BEAT_AGENTS]="" + VARS[VITE_DEFAULT_NUDGERS]="" ;; base-sepolia) VARS[COMMONALITY_ENVIRONMENT]="testnet" VARS[CHAIN_ID]="84532" VARS[BASE_SEPOLIA_RPC_URL]="${VARS[BASE_SEPOLIA_RPC_URL]:-https://sepolia.base.org}" - VARS[ETHEREUM_RPC_URL]="${VARS[ETHEREUM_RPC_URL]:-${VARS[BASE_SEPOLIA_RPC_URL]}}" + VARS[ETHEREUM_RPC_URL]="${VARS[BASE_SEPOLIA_RPC_URL]}" VARS[IPFS_API]="${VARS[IPFS_API]:-https://ipfs.io/api/v0}" VARS[IPFS_GATEWAY]="${VARS[IPFS_GATEWAY]:-https://ipfs.io/ipfs}" VARS[EVENT_CACHE_URL]="${VARS[EVENT_CACHE_URL]:-https://commonality-indexer.onrender.com}" @@ -165,7 +169,7 @@ mainnet) VARS[COMMONALITY_ENVIRONMENT]="mainnet" VARS[CHAIN_ID]="1" VARS[MAINNET_RPC_URL]="${VARS[MAINNET_RPC_URL]:-https://eth.llamarpc.com}" - VARS[ETHEREUM_RPC_URL]="${VARS[ETHEREUM_RPC_URL]:-${VARS[MAINNET_RPC_URL]}}" + VARS[ETHEREUM_RPC_URL]="${VARS[MAINNET_RPC_URL]}" VARS[IPFS_API]="${VARS[IPFS_API]:-https://ipfs.io/api/v0}" VARS[IPFS_GATEWAY]="${VARS[IPFS_GATEWAY]:-https://ipfs.io/ipfs}" VARS[BASE_PAYMASTER_URL]="${VARS[BASE_PAYMASTER_URL]:-${VARS[PLATFORM_API_URL]:-}/sponsored-gas/paymaster}" @@ -176,6 +180,16 @@ mainnet) ;; esac +if [ "$NETWORK" != "localhost" ]; then + for key in ETHEREUM_RPC_URL EVENT_CACHE_URL PLATFORM_API_URL IPFS_GATEWAY; do + value="${VARS[$key]:-}" + if [[ "$value" =~ ^https?://(localhost|127\.0\.0\.1)(:|/|$) ]]; then + echo "Error: refusing mixed $NETWORK profile: $key points at $value" >&2 + exit 1 + fi + done +fi + populate_ui_domain_urls # ============================================================ @@ -267,7 +281,11 @@ echo " wrote $ROOT/integration-tests/.env.local" echo "VITE_PRIVY_SMART_WALLET_BUNDLER_URL=${VARS[BASE_SEPOLIA_BUNDLER_URL]:-}" echo "VITE_PRIVY_SMART_WALLET_PAYMASTER_URL=${VARS[BASE_SEPOLIA_PAYMASTER_URL]:-}" fi - echo "VITE_EVENT_CACHE_URL=${VARS[EVENT_CACHE_URL]:-}" + if [ -n "${VARS[UI_EVENT_CACHE_URL]+x}" ]; then + echo "VITE_EVENT_CACHE_URL=${VARS[UI_EVENT_CACHE_URL]}" + else + echo "VITE_EVENT_CACHE_URL=${VARS[EVENT_CACHE_URL]:-}" + fi echo "VITE_DISPLAY_DENYLIST_URL=${VARS[DISPLAY_DENYLIST_URL]:-}" echo "VITE_POLICY_BUNDLE_URL=${VARS[POLICY_BUNDLE_URL]:-}" echo "VITE_ETH_RPC_URL=${VARS[ETHEREUM_RPC_URL]:-}" @@ -323,5 +341,13 @@ echo " wrote $ROOT/ui/.env" cp "$ROOT/ui/.env" "$ROOT/causestarter/.env" echo " wrote $ROOT/causestarter/.env" +PROFILE_DIR="$ROOT/.generated-env/$NETWORK" +mkdir -p "$PROFILE_DIR" +cp "$ROOT/.env" "$PROFILE_DIR/root.env" +cp "$ROOT/integration-tests/.env.local" "$PROFILE_DIR/integration-tests.env.local" +cp "$ROOT/ui/.env" "$PROFILE_DIR/ui.env" +cp "$ROOT/causestarter/.env" "$PROFILE_DIR/causestarter.env" +echo " saved complete profile $PROFILE_DIR" + echo "" echo "Done! Environment configured for network: $NETWORK" diff --git a/workflow/deployment.md b/workflow/deployment.md index ff33a5e6..26685607 100644 --- a/workflow/deployment.md +++ b/workflow/deployment.md @@ -221,6 +221,11 @@ Run `./scripts/setup-env.sh base-sepolia` after wallet generation. It publishes the chain-scoped `VITE_DEFAULT_ALIGNMENT_TRUST_ROOT` derived from the dedicated bootstrap key into both the domain UI and CauseStarter Vite configuration. +Each generated network is also snapshotted under gitignored +`.generated-env//`. UI publish scripts activate the requested network +only for their build/upload and restore the complete localhost profile on exit, +including failed publishes. + The IPFS UI cannot use the local Vite proxy, so this URL is written into `ui/.env` by `scripts/setup-env.sh` and emitted into each domain's runtime `config.json` by the Vite build. `scripts/deploy-ui.sh` will stop early if `VITE_EVENT_CACHE_URL` is missing. #### How the naming layer works (testnet) diff --git a/workflow/local-development.md b/workflow/local-development.md index e60a354c..c5b6cc5a 100644 --- a/workflow/local-development.md +++ b/workflow/local-development.md @@ -26,6 +26,11 @@ After building, you can run: That's it. This uses Docker Compose to start a local Hardhat blockchain, deploys the smart contracts, starts IPFS, the Ponder indexer, and the platform API service, then publishes the selected UI domain SPA(s) to the local IPFS gateway. +`services.sh --start` first regenerates and activates the complete `localhost` +profile (chain 31337, local RPC/indexer, and local contracts), so a prior +testnet UI build cannot leak Base Sepolia settings into the local stack. +Generated snapshots live under gitignored `.generated-env//`. + **Which UI bundles get built:** local start currently publishes **CauseStarter only**. The eight legacy `ui` domains (commonality, lazyGiving, alignment, tally, content-funding, civility, common-sense-majority, conceptspace) each run a full Docker Vite build sequentially and were a major part of `--start` time. This is a temporary, reversible default — the compose services and source trees are still there. - Default: `LOCAL_UI_DOMAINS=causestarter` (implicit) From 4bb81029f1ba907fce78566f5b34573808870711 Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 16:15:24 -0400 Subject: [PATCH 17/19] Record local campaign UI inspection on chain 31337. Restarted a stale CauseStarter Vite server so it stopped hitting Base Sepolia, walked representative boards/statements/projects, and closed plan item 7. --- CONTINUITY.md | 6 +++- .../TESTNET-SIMULATION-PLAN.md | 2 +- .../medium-realistic-v1-local-run.md | 30 +++++++++++++++---- workflow/local-development.md | 3 ++ 4 files changed, 33 insertions(+), 8 deletions(-) diff --git a/CONTINUITY.md b/CONTINUITY.md index 1fcfd1f2..08b8bba5 100644 --- a/CONTINUITY.md +++ b/CONTINUITY.md @@ -2158,4 +2158,8 @@ Item 7 of the medium-realistic campaign: `campaignProvisioning.ts` funds generat ## 2026-09-11 — Full local 100-user campaign execute + reconcile -Ran `gen:campaign:execute --mode local --concurrency 4 --skip-provision`: 1932 mined, 0 failed, ~196s. First reconcile mismatched all fund-project and note SDK folds. Fixes: derived funding checks now use `campaignFundProjectCost()` (0.01 token per buy, matching the adapter) instead of planner persona amounts; note lookups lowercase the bound contract address. Re-reconcile: 1932/1932 verified, indexer lag 0. CauseStarter UI on this host still talks to Base Sepolia (`chainId=84532`), so browser inspection of campaign boards is blocked until the UI points at 31337. Notes: `fake-data-generation/campaigns/medium-realistic-v1-local-run.md`. Plan item 7 remains open for that UI pass. +Ran `gen:campaign:execute --mode local --concurrency 4 --skip-provision`: 1932 mined, 0 failed, ~196s. First reconcile mismatched all fund-project and note SDK folds. Fixes: derived funding checks now use `campaignFundProjectCost()` (0.01 token per buy, matching the adapter) instead of planner persona amounts; note lookups lowercase the bound contract address. Re-reconcile: 1932/1932 verified, indexer lag 0. + +## 2026-09-11 — Local campaign CauseStarter UI inspection + +Plan item 7 closed. Restarted a 6-day-old `causestarter:dev` so Vite baked `VITE_CHAIN_ID=31337` instead of Base Sepolia. Schools and open-source boards, a bridge statement, and a funded project (eip155:31337) render from the local indexer. Notes page needs a connected wallet. Empty `VITE_DEFAULT_ALIGNMENT_TRUST_ROOT` shows the “no starter vouching network” banner. Details: `fake-data-generation/campaigns/medium-realistic-v1-local-run.md`. Next: remote canary preflight (plan item 8). diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index e69aa2ea..27000288 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -183,7 +183,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 4. **[x] Add resumable, budgeted execution.** [`campaignExecutor.ts`](./campaignExecutor.ts) atomically persists planned/submitted/mined/failed action state and transaction hashes behind a chain-adapter boundary. It resumes submitted transactions through receipt lookup, schedules only dependency-ready actions, serializes budget reservation while allowing bounded receipt concurrency, and adds pacing, classified retries/backoff, cooperative stop/resume, transaction caps, and fail-closed native-token budgets. Fake-chain tests prove interruption/resume without duplicate submission, retry behavior, immutable-plan matching, and budget refusal; a later campaign action adapter will bind this reusable layer to contracts. 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. 6. **[x] Bind execution to contracts and deep reconciliation.** [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) turns planned actions into contract writes, records mined hashes, and updates the public runtime-binding artifact. `npm run gen:campaign:execute` is local-first, refuses remote mutation without `--confirm-remote-mutation`, and keeps wallet secrets outside the campaign directory. Reconciliation already maps those hashes to Ponder events and SDK folds. Local execution now funds generated wallets (ETH plus payment tokens, with mint fallback) and writes `execution/funding-ledger.json` before submitting writes. -7. **[ ] Run and inspect the full 100-user campaign locally.** Local execute mined all 1932 actions in ~196s (concurrency 4); after harness fixes, reconciliation is 1932/1932 with lag 0. See [`campaigns/medium-realistic-v1-local-run.md`](./campaigns/medium-realistic-v1-local-run.md). Remaining: inspect representative CauseStarter pages against a UI pointed at local chain 31337 (current Vite/IPFS bundles were on Base Sepolia). +7. **[x] Run and inspect the full 100-user campaign locally.** Local execute mined all 1932 actions in ~196s (concurrency 4); after harness fixes, reconciliation is 1932/1932 with lag 0. CauseStarter on Vite `:5174` (chain 31337, indexer via `/api` → `:42069`) shows campaign boards, statements, and funded projects. See [`campaigns/medium-realistic-v1-local-run.md`](./campaigns/medium-realistic-v1-local-run.md). A Vite process started against a previous testnet env will keep hitting Base Sepolia until it is restarted. 8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. 10. **[ ] Run staged 25-user then 100-user campaigns.** Re-run health/preflight and reconcile after each phase. Do not automatically advance after a failed or materially surprising phase. diff --git a/fake-data-generation/campaigns/medium-realistic-v1-local-run.md b/fake-data-generation/campaigns/medium-realistic-v1-local-run.md index ea1e4a63..1e747f4d 100644 --- a/fake-data-generation/campaigns/medium-realistic-v1-local-run.md +++ b/fake-data-generation/campaigns/medium-realistic-v1-local-run.md @@ -20,15 +20,33 @@ First pass reported `derived-mismatch` for every `fund-project` and note action After those harness fixes, `npm run gen:campaign:reconcile` reported **1932/1932 verified**, no missing/duplicate indexed events. -## UI inspection +## UI inspection (2026-09-11, Vite `:5174` after restart) CauseStarter has no global cause directory; campaign boards are reachable only by organizer URL: -`/cause//` from `execution/runtime-bindings.json` (example: schools-common-ground owner `0x29Aad1ae4EC538790a3231c62d84d6840685D613`, ref `campaign-medium-realistic-v1-schools-common-ground`). +`/cause//` from `execution/runtime-bindings.json`. Vite uses path routing (`http://localhost:5174/cause/...`). The IPFS bundle still uses hash routing (`http://causestarter.localhost:8088/#/cause/...`). -On this machine the Vite app (`:5174`) and the published IPFS bundle (`http://causestarter.localhost:8088/#/`) both requested `chainId=84532` at `https://commonality-indexer.onrender.com`. Cause pages showed “Failed to fetch” (CORS). That is the root `.env` / `causestarter/.env` pointing at Base Sepolia, not a campaign indexing omission. Re-inspect after pointing the UI at chain 31337 and `http://localhost:42069`. +A Vite process that had been running since 2026-09-05 still served a Base Sepolia `import.meta.env` (`chainId=84532`, `https://commonality-indexer.onrender.com`). Cause pages showed “Failed to fetch” (CORS). `ui/.env` already had `VITE_CHAIN_ID=31337`; restarting `npm run causestarter:dev` picked it up. Event cache with empty `VITE_EVENT_CACHE_URL` uses `window.location.origin` and Vite’s `/api` proxy to `http://localhost:42069`. -## Still open for plan item 7 +Representative pages (no wallet connected): -- Browser pass against a local-chain UI (cause board, a funded project, a statement, a note/delegate path). -- Optional: make `fund-project` writes use planned persona amounts (would require a wipe/re-execute so totals stay reconcilable). +| Surface | URL | What showed | +|---|---|---| +| Schools cause board | `/cause/0x29Aad1ae4EC538790a3231c62d84d6840685D613/campaign-medium-realistic-v1-schools-common-ground` | SYNTHETIC TESTNET CAMPAIGN label; 5 statements with mixed 0–8 support; 4 funding projects; raised **3.09 / 8** USDZZZ; organizer checksum | +| Open-source cause board | `/cause/0x29Aad1ae4EC538790a3231c62d84d6840685D613/campaign-medium-realistic-v1-open-source` | Uneven vs schools: 21 signed at least one, 2 signed all; 1 project **0.47 / 2** USDZZZ | +| Bridge statement | `/statement/bafkreihzjxdzmdf7jeaqwkfz5c6agjaqimsd7h6ozrlsux6eilazdpklzu?mode=sign` | Full accepted text; **8 · 4 direct · 7 indirect** | +| Funded project | `/projects/eip155%3A31337%3A0x9bd03768a7dcc129555de410ff8e85528a4f88b5` | **1.15 / 2** USDZZZ, 29d left, contributor table, two statement vouches, giving options 0.1 / 0.05 / 0.01 | +| Delegated funds | `/delegation/notes` | Connect-wallet empty state (expected without a campaign wallet in the browser) | + +Product / ops observations (not indexer omissions): + +- Boards are labelled synthetic; titles are slug-like (`schools-common-ground`) and project cards repeat long statement text. +- `VITE_DEFAULT_ALIGNMENT_TRUST_ROOT` is empty in the local generated profile, so the cause page warns that the starter vouching network is unavailable. Projects still listed (direct vouches on the cards). +- Coherence badge stayed “not confirmed”. +- Cause-assist on `:3002` was down; Vite logged `/health` 500s. Did not block reads. +- Notes/delegations are not inspectable without connecting a campaign wallet. + +## Still optional after item 7 + +- Make `fund-project` writes use planned persona amounts (would require a wipe/re-execute so totals stay reconcilable). +- Connect a campaign wallet to walk a note/delegate path in the UI. diff --git a/workflow/local-development.md b/workflow/local-development.md index c5b6cc5a..6343a3f4 100644 --- a/workflow/local-development.md +++ b/workflow/local-development.md @@ -30,6 +30,9 @@ That's it. This uses Docker Compose to start a local Hardhat blockchain, deploys profile (chain 31337, local RPC/indexer, and local contracts), so a prior testnet UI build cannot leak Base Sepolia settings into the local stack. Generated snapshots live under gitignored `.generated-env//`. +Restart `npm run causestarter:dev` after that switch: Vite bakes `VITE_*` at +process start, so a leftover server will keep calling the previous network +(for example Base Sepolia indexer from a localhost profile). **Which UI bundles get built:** local start currently publishes **CauseStarter only**. The eight legacy `ui` domains (commonality, lazyGiving, alignment, tally, content-funding, civility, common-sense-majority, conceptspace) each run a full Docker Vite build sequentially and were a major part of `--start` time. This is a temporary, reversible default — the compose services and source trees are still there. From f5e16bb4e78dbca05ef659e15004e7514e63453c Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 16:22:59 -0400 Subject: [PATCH 18/19] Add a read-only remote canary preflight for the 10-user campaign. Slice the frozen plan, estimate funding and duration, check chain and indexer lag, and write an approval proposal without mutating testnet. --- CONTINUITY.md | 4 + fake-data-generation/README.md | 2 + .../TESTNET-SIMULATION-PLAN.md | 2 +- fake-data-generation/campaignCanary.ts | 331 ++++++++++++++++++ fake-data-generation/package.json | 1 + .../test/campaignCanary.test.ts | 129 +++++++ 6 files changed, 468 insertions(+), 1 deletion(-) create mode 100644 fake-data-generation/campaignCanary.ts create mode 100644 fake-data-generation/test/campaignCanary.test.ts diff --git a/CONTINUITY.md b/CONTINUITY.md index 08b8bba5..442e921c 100644 --- a/CONTINUITY.md +++ b/CONTINUITY.md @@ -2163,3 +2163,7 @@ Ran `gen:campaign:execute --mode local --concurrency 4 --skip-provision`: 1932 m ## 2026-09-11 — Local campaign CauseStarter UI inspection Plan item 7 closed. Restarted a 6-day-old `causestarter:dev` so Vite baked `VITE_CHAIN_ID=31337` instead of Base Sepolia. Schools and open-source boards, a bridge statement, and a funded project (eip155:31337) render from the local indexer. Notes page needs a connected wallet. Empty `VITE_DEFAULT_ALIGNMENT_TRUST_ROOT` shows the “no starter vouching network” banner. Details: `fake-data-generation/campaigns/medium-realistic-v1-local-run.md`. Next: remote canary preflight (plan item 8). + +## 2026-09-11 — Remote canary preflight (no mutation) + +Plan item 8 closed. `campaignCanary.ts` + `npm run gen:campaign:canary-preflight` slice the medium-realistic plan to 10 users (plus extra actors required by prerequisites), estimate ETH/token needs and paced duration, probe chain bytecode and indexer lag, and write `reports/remote-canary-preflight.{json,md}`. Shared-lab health, official implication path, `verifier-testnet.sh`, and the budget/window remain `needs-adam`. Does not send transactions. Next: item 9 only after Adam approves that proposal. diff --git a/fake-data-generation/README.md b/fake-data-generation/README.md index e5aa5e0a..c8df8756 100644 --- a/fake-data-generation/README.md +++ b/fake-data-generation/README.md @@ -15,6 +15,8 @@ The medium realistic campaign has a separate planning-only command: `npm run gen Campaign execution uses the explicit environment boundary in `campaignEnvironment.ts`. Local mode is pinned to Hardhat chain 31337 and may use the legacy deploy/Hardhat-wallet/free-mint conveniences. Remote mode requires a committed deployment env manifest, an explicit non-Hardhat chain ID, existing bytecode at every required address, generated campaign wallets, and transfer-only payment-token provisioning. Deployment and provisioning adapters are validated against those policies before they can run; remote execution must also carry a separate mutation-confirmation flag when it is added. The existing `gen:tiny`/`gen:small`/`gen:medium`/`gen:large` commands remain local-only legacy simulation paths and are not remote campaign commands. +Remote canary prep is `npm run gen:campaign:canary-preflight`. It is read-only: chain/bytecode + indexer lag, a 10-user slice of the planned graph, native/token budget and duration estimates, and secrets/retention notes. It writes `reports/remote-canary-preflight.json` and `.md` for Adam's approval and never sets `--confirm-remote-mutation`. Do not run remote execute until that proposal is approved. + The reusable runner in `campaignExecutor.ts` consumes a frozen action plan through a chain adapter. It atomically persists planned/submitted/mined/failed state and transaction hashes, resumes submitted transactions by receipt lookup, respects action prerequisites, and enforces bounded concurrency, pacing, classified retries, a transaction cap, and a native-token budget. [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) is the contract-binding layer: it maps each planned action type onto SDK writes, classifies RPC vs revert failures, and updates public runtime bindings (no private keys). `npm run gen:campaign:execute` runs that adapter locally after environment preflight and funds generated wallets (ETH plus payment tokens, minting locally when transfer fails). Remote mode still requires `--confirm-remote-mutation` and stays transfer-only. Pass `--skip-provision` only when wallets are already funded. Generated wallet secrets stay under `output/campaigns/secrets/` (gitignored via `output/`). The funding ledger is `execution/funding-ledger.json`. Each local `fund-project` write currently spends `CAMPAIGN_FUND_PROJECT_TOKEN` (`0.01` of the 6-decimal payment token), not the planner’s persona-sized `amount`. Reconciliation’s SDK funding check uses that same unit cost. Note folds are looked up as `:`. diff --git a/fake-data-generation/TESTNET-SIMULATION-PLAN.md b/fake-data-generation/TESTNET-SIMULATION-PLAN.md index 27000288..538837d9 100644 --- a/fake-data-generation/TESTNET-SIMULATION-PLAN.md +++ b/fake-data-generation/TESTNET-SIMULATION-PLAN.md @@ -184,7 +184,7 @@ Do these in order unless Adam names a different item. Keep each item small enoug 5. **[x] Make simulated behavior persona- and cause-aware.** The deterministic planner now emits concrete belief values and linked belief changes, readable projects derived from their accepted outcome statements, outcome-backed alignments, cause-member funding with persona-sized amounts and deliberately skewed project popularity (including unfunded projects), and note delegations restricted to trusted-role users who share a cause with the donor. Implication actions remain limited to accepted bridge-role evidence. Planner validation and tests enforce the behavioral payloads, relationships, and histories before execution. 6. **[x] Bind execution to contracts and deep reconciliation.** [`campaignActionAdapter.ts`](./campaignActionAdapter.ts) turns planned actions into contract writes, records mined hashes, and updates the public runtime-binding artifact. `npm run gen:campaign:execute` is local-first, refuses remote mutation without `--confirm-remote-mutation`, and keeps wallet secrets outside the campaign directory. Reconciliation already maps those hashes to Ponder events and SDK folds. Local execution now funds generated wallets (ETH plus payment tokens, with mint fallback) and writes `execution/funding-ledger.json` before submitting writes. 7. **[x] Run and inspect the full 100-user campaign locally.** Local execute mined all 1932 actions in ~196s (concurrency 4); after harness fixes, reconciliation is 1932/1932 with lag 0. CauseStarter on Vite `:5174` (chain 31337, indexer via `/api` → `:42069`) shows campaign boards, statements, and funded projects. See [`campaigns/medium-realistic-v1-local-run.md`](./campaigns/medium-realistic-v1-local-run.md). A Vite process started against a previous testnet env will keep hitting Base Sepolia until it is restarted. -8. **[ ] Prepare the remote canary.** Calculate funding, provider limits, expected duration, secrets handling, test-data labelling, and retention. Add a preflight that enforces the readiness gate and outputs the exact proposed mutation/budget for Adam's approval. +8. **[x] Prepare the remote canary.** [`campaignCanary.ts`](./campaignCanary.ts) slices the frozen plan to 10 users plus dependency actors, estimates native/token funding and paced duration, and writes a read-only proposal (`reports/remote-canary-preflight.{json,md}`). `npm run gen:campaign:canary-preflight` checks remote chain ID/bytecode and indexer lag, refuses `--confirm-remote-mutation`, documents secrets layout/retention and the synthetic label, and leaves shared-lab, official implication path, verifier-testnet, and budget-window items as `needs-adam`. It does not send transactions. 9. **[ ] Run 10 users on testnet and reconcile.** Stop on unexplained discrepancies. Demonstrate safe resume and confirm the populated pages are readable before expanding. 10. **[ ] Run staged 25-user then 100-user campaigns.** Re-run health/preflight and reconcile after each phase. Do not automatically advance after a failed or materially surprising phase. 11. **[ ] Publish the campaign report and follow-up decisions.** Record product findings, operational limits, indexing correctness/latency, gas/provider costs, and the explicitly unsupported scalability claims. Move concrete fixes to the appropriate backlog or a new focus; do not let this plan become a permanent catch-all. diff --git a/fake-data-generation/campaignCanary.ts b/fake-data-generation/campaignCanary.ts new file mode 100644 index 00000000..39e2d815 --- /dev/null +++ b/fake-data-generation/campaignCanary.ts @@ -0,0 +1,331 @@ +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { formatEther, type Address } from 'viem'; +import { + BASE_SEPOLIA_CHAIN_ID, + createCampaignChainAdapter, + loadCampaignEnvironment, + preflightCampaignEnvironment, + type CampaignChainAdapter, + type CampaignEnvironment, + type CampaignPreflightResult, +} from './campaignEnvironment.js'; +import { createLocalCampaignStack, probeLocalCampaignStack } from './campaignLocalStack.js'; +import { loadCampaignPlan, type CampaignPlan, type PlannedAction } from './campaignPlanner.js'; +import { computeCampaignFundingNeeds } from './campaignProvisioning.js'; +import type { CampaignManifestV1 } from './campaignSchema.js'; +import { loadEnv, RPC_URL } from './loadEnv.js'; + +export const REMOTE_CANARY_USER_COUNT = 10; +export const DEFAULT_REMOTE_PACING_MS = 250; +export const DEFAULT_REMOTE_CONCURRENCY = 1; +export const DEFAULT_MAX_INDEXER_LAG_BLOCKS = 300n; +export const DEFAULT_TESTNET_INDEXER_URL = 'https://commonality-indexer.onrender.com'; +export const DEFAULT_TESTNET_DEPLOYMENT_ENV = '../deployments/base-sepolia.env'; + +export const CANARY_SECRETS_POLICY = { + storage: 'output/campaigns/secrets/ (gitignored; never under the public campaign directory)', + walletSource: 'generated-only', + hardhatKeys: 'forbidden on remote', + retention: 'delete after the campaign report unless Adam explicitly retains them', + identification: 'SYNTHETIC TESTNET CAMPAIGN — NOT REAL USERS OR ADOPTION', +} as const; + +export interface CampaignCanarySlice { + userCount: number; + users: CampaignPlan['users']; + extraActors: CampaignPlan['users']; + actions: PlannedAction[]; + projects: CampaignPlan['projects']; + writesByType: Record; +} + +export interface CampaignCanaryGate { + id: string; + status: 'pass' | 'fail' | 'needs-adam'; + detail: string; +} + +export interface CampaignCanaryProposal { + campaignId: string; + phase: 'remote-canary-10'; + mutatesChain: false; + userCount: number; + writeCount: number; + writesByType: Record; + nativeWeiNeeded: string; + nativeEthNeeded: string; + paymentTokenUnitsNeeded: string; + pacingMs: number; + concurrency: number; + estimatedDurationMs: number; + estimatedDurationMinutes: number; + nativeTokenBudgetWei: string; + transactionCap: number; + secrets: typeof CANARY_SECRETS_POLICY; + syntheticDataLabel: string; + chainId: number; + contracts: CampaignEnvironment['contracts']; + indexerUrl: string; + indexerLagBlocks: string | null; + gates: CampaignCanaryGate[]; + readyForAdamApproval: boolean; +} + +export function sliceCampaignPlanForCanary(plan: CampaignPlan, userCount = REMOTE_CANARY_USER_COUNT): CampaignCanarySlice { + if (!Number.isSafeInteger(userCount) || userCount < 1) throw new Error('canary user count must be a positive integer'); + if (userCount > plan.users.length) throw new Error(`canary needs ${userCount} users but the plan has ${plan.users.length}`); + const users = plan.users.slice(0, userCount); + const userIds = new Set(users.map((user) => user.id)); + const byId = new Map(plan.actions.map((action) => [action.id, action])); + const selected = new Set(); + const addWithDeps = (actionId: string): void => { + if (selected.has(actionId)) return; + const action = byId.get(actionId); + if (!action) throw new Error(`canary slice is missing action ${actionId}`); + for (const dependencyId of action.dependsOn) addWithDeps(dependencyId); + selected.add(actionId); + }; + for (const action of plan.actions) { + if (action.actorUserId === null || (action.actorUserId !== null && userIds.has(action.actorUserId))) addWithDeps(action.id); + if (action.delegateUserId && userIds.has(action.delegateUserId)) addWithDeps(action.id); + } + const actions = plan.actions.filter((action) => selected.has(action.id)); + const actorIds = new Set(actions.flatMap((action) => (action.actorUserId ? [action.actorUserId] : []))); + const extraActors = plan.users.filter((user) => actorIds.has(user.id) && !userIds.has(user.id)); + const projectIds = new Set(actions.flatMap((action) => (action.projectId ? [action.projectId] : []))); + const projects = plan.projects.filter((project) => projectIds.has(project.id) || userIds.has(project.founderUserId)); + const writesByType: Record = {}; + for (const action of actions) writesByType[action.type] = (writesByType[action.type] ?? 0) + 1; + return { userCount, users, extraActors, actions, projects, writesByType }; +} + +function placeholderWallets(users: CampaignPlan['users']) { + return users.map((user, index) => ({ + walletSlot: user.walletSlot, + address: `0x${String(index + 1).padStart(40, '0')}` as Address, + privateKey: `0x${String(index + 1).padStart(64, '0')}` as `0x${string}`, + source: 'generated' as const, + })); +} + +export function estimateCanaryDurationMs(writeCount: number, pacingMs: number, concurrency: number): number { + const safeConcurrency = Math.max(1, concurrency); + const batches = Math.ceil(writeCount / safeConcurrency); + return batches * Math.max(0, pacingMs); +} + +function passFail(ok: boolean): CampaignCanaryGate['status'] { + return ok ? 'pass' : 'fail'; +} + +function remoteCanaryGates(input: { + environment: CampaignEnvironment; + chainPreflight: CampaignPreflightResult; + slice: CampaignCanarySlice; + secretsPath: string; + indexerLagBlocks: bigint | null; + maxLag: bigint; +}): CampaignCanaryGate[] { + const lagOk = input.indexerLagBlocks !== null && input.indexerLagBlocks <= input.maxLag; + return [ + { id: 'remote-mode', status: 'pass', detail: `mode ${input.environment.mode}; chain ${input.chainPreflight.chainId}` }, + { id: 'existing-deployment', status: passFail(input.environment.deployment.strategy === 'existing-only'), detail: input.environment.deployment.strategy }, + { id: 'transfer-only-tokens', status: passFail(input.environment.provisioning.paymentTokenStrategy === 'transfer-only'), detail: input.environment.provisioning.paymentTokenStrategy }, + { id: 'generated-wallets', status: passFail(input.environment.provisioning.walletSource === 'generated-only'), detail: input.environment.provisioning.walletSource }, + { id: 'bytecode', status: passFail(input.chainPreflight.checkedContracts.length > 0), detail: input.chainPreflight.checkedContracts.join(',') }, + { id: 'no-mutation-in-preflight', status: passFail(input.environment.mode === 'remote' && !input.environment.mutationConfirmed), detail: 'preflight must not set --confirm-remote-mutation' }, + { id: 'canary-size', status: passFail(input.slice.userCount === REMOTE_CANARY_USER_COUNT), detail: `${input.slice.userCount} users, ${input.slice.actions.length} writes` }, + { id: 'secrets-layout', status: passFail(input.secretsPath.startsWith('../secrets/')), detail: input.secretsPath }, + { + id: 'indexer-lag', + status: passFail(lagOk), + detail: input.indexerLagBlocks === null ? 'indexer head unavailable' : `${input.indexerLagBlocks} blocks (max ${input.maxLag})`, + }, + { id: 'shared-lab-readiness', status: 'needs-adam', detail: 'Confirm workflow/testnet-working-plan.md shared-lab milestone is boring before mutating' }, + { id: 'official-implication-path', status: 'needs-adam', detail: 'Confirm the official implication/trust path for the chosen statements is intentionally populated' }, + { id: 'read-only-verifier', status: 'needs-adam', detail: 'Run ./scripts/verifier-testnet.sh (read-only leaves) immediately before the mutating canary' }, + { id: 'budget-and-window', status: 'needs-adam', detail: 'Adam must approve this proposal, native budget, pacing, and testnet window' }, + ]; +} + +export function buildRemoteCanaryProposal(input: { + plan: CampaignPlan; + manifest: CampaignManifestV1; + environment: CampaignEnvironment; + chainPreflight: CampaignPreflightResult; + slice?: CampaignCanarySlice; + pacingMs?: number; + concurrency?: number; + indexerUrl: string; + indexerLagBlocks: bigint | null; + maxIndexerLagBlocks?: bigint; +}): CampaignCanaryProposal { + if (input.environment.mode !== 'remote') throw new Error('remote canary preflight refuses local mode'); + const slice = input.slice ?? sliceCampaignPlanForCanary(input.plan); + const pacingMs = input.pacingMs ?? DEFAULT_REMOTE_PACING_MS; + const concurrency = input.concurrency ?? DEFAULT_REMOTE_CONCURRENCY; + const fundedUsers = [...slice.users, ...slice.extraActors]; + const slicedPlan = { ...input.plan, users: fundedUsers, actions: slice.actions, projects: slice.projects }; + const needs = computeCampaignFundingNeeds(slicedPlan, placeholderWallets(fundedUsers)); + const nativeWei = needs.reduce((sum, need) => sum + need.nativeWei, 0n); + const paymentTokenUnits = needs.reduce((sum, need) => sum + need.paymentTokenUnits, 0n); + const estimatedDurationMs = estimateCanaryDurationMs(slice.actions.length, pacingMs, concurrency); + const maxLag = input.maxIndexerLagBlocks ?? DEFAULT_MAX_INDEXER_LAG_BLOCKS; + const gates = remoteCanaryGates({ + environment: input.environment, + chainPreflight: input.chainPreflight, + slice, + secretsPath: input.manifest.artifactLayout.walletSecrets, + indexerLagBlocks: input.indexerLagBlocks, + maxLag, + }); + const readyForAdamApproval = gates.every((gate) => gate.status !== 'fail'); + return { + campaignId: input.plan.campaignId, + phase: 'remote-canary-10', + mutatesChain: false, + userCount: slice.userCount, + writeCount: slice.actions.length, + writesByType: slice.writesByType, + nativeWeiNeeded: nativeWei.toString(), + nativeEthNeeded: formatEther(nativeWei), + paymentTokenUnitsNeeded: paymentTokenUnits.toString(), + pacingMs, + concurrency, + estimatedDurationMs, + estimatedDurationMinutes: Math.round((estimatedDurationMs / 60_000) * 10) / 10, + nativeTokenBudgetWei: nativeWei.toString(), + transactionCap: slice.actions.length, + secrets: CANARY_SECRETS_POLICY, + syntheticDataLabel: input.manifest.campaign.syntheticDataLabel, + chainId: input.chainPreflight.chainId, + contracts: input.environment.contracts, + indexerUrl: input.indexerUrl, + indexerLagBlocks: input.indexerLagBlocks === null ? null : input.indexerLagBlocks.toString(), + gates, + readyForAdamApproval, + }; +} + +export function formatCanaryProposalMarkdown(proposal: CampaignCanaryProposal): string { + const gateLines = proposal.gates.map((gate) => `- [${gate.status}] ${gate.id}: ${gate.detail}`).join('\n'); + const typeLines = Object.entries(proposal.writesByType).map(([type, count]) => `- ${type}: ${count}`).join('\n'); + return `# Remote canary proposal (${proposal.campaignId}) + +This preflight **does not mutate** the chain. Do not run \`gen:campaign:execute --mode remote --confirm-remote-mutation\` until Adam approves this file. + +- Users: ${proposal.userCount} +- Writes: ${proposal.writeCount} +- Native needed: ${proposal.nativeEthNeeded} ETH (${proposal.nativeWeiNeeded} wei) +- Payment-token units: ${proposal.paymentTokenUnitsNeeded} +- Pacing: ${proposal.pacingMs} ms, concurrency ${proposal.concurrency} +- Estimated duration: ${proposal.estimatedDurationMinutes} minutes +- Transaction cap: ${proposal.transactionCap} +- Chain ID: ${proposal.chainId} +- Indexer: ${proposal.indexerUrl} (lag ${proposal.indexerLagBlocks ?? 'unknown'} blocks) +- Synthetic label: ${proposal.syntheticDataLabel} +- Secrets: ${proposal.secrets.storage}; ${proposal.secrets.retention} + +## Writes by type + +${typeLines} + +## Readiness gates + +${gateLines} + +Ready for Adam approval: **${proposal.readyForAdamApproval ? 'yes (pending the needs-adam items)' : 'no'}** +`; +} + +export async function runRemoteCanaryPreflight(input: { + manifest: CampaignManifestV1; + plan: CampaignPlan; + environment: CampaignEnvironment; + chain: CampaignChainAdapter; + indexerUrl: string; + outputDirectory: string; + pacingMs?: number; + concurrency?: number; + probeIndexer?: () => Promise<{ chainHead: bigint; indexerHead: bigint }>; +}): Promise<{ proposal: CampaignCanaryProposal; jsonPath: string; markdownPath: string }> { + if (input.environment.mode !== 'remote') throw new Error('remote canary preflight refuses local mode'); + const chainPreflight = await preflightCampaignEnvironment(input.environment, input.chain); + const probe = input.probeIndexer ?? (async () => { + const stack = createLocalCampaignStack({ + rpcUrl: input.environment.rpcUrl, + eventCacheUrl: input.indexerUrl, + chainId: input.environment.expectedChainId, + }); + return probeLocalCampaignStack(stack); + }); + let indexerLagBlocks: bigint | null = null; + try { + const heads = await probe(); + indexerLagBlocks = heads.chainHead > heads.indexerHead ? heads.chainHead - heads.indexerHead : 0n; + } catch { + indexerLagBlocks = null; + } + const proposal = buildRemoteCanaryProposal({ + plan: input.plan, + manifest: input.manifest, + environment: input.environment, + chainPreflight, + pacingMs: input.pacingMs, + concurrency: input.concurrency, + indexerUrl: input.indexerUrl, + indexerLagBlocks, + }); + const jsonPath = path.join(input.outputDirectory, 'reports/remote-canary-preflight.json'); + const markdownPath = path.join(input.outputDirectory, 'reports/remote-canary-preflight.md'); + await mkdir(path.dirname(jsonPath), { recursive: true }); + await writeFile(jsonPath, `${JSON.stringify(proposal, null, 2)}\n`); + await writeFile(markdownPath, formatCanaryProposalMarkdown(proposal)); + return { proposal, jsonPath, markdownPath }; +} + +loadEnv(); + +function parseOption(name: string, fallback?: string): string | undefined { + const index = process.argv.indexOf(name); + if (index >= 0 && process.argv[index + 1]) return process.argv[index + 1]; + return fallback; +} + +async function main(): Promise { + const directory = path.dirname(fileURLToPath(import.meta.url)); + const manifestPath = parseOption('--manifest', path.join(directory, 'campaigns/medium-realistic-v1.json'))!; + const outputDirectory = parseOption('--output', path.join(directory, 'output/campaigns/medium-realistic-v1'))!; + const deploymentEnvPath = parseOption('--deployment-env', path.join(directory, DEFAULT_TESTNET_DEPLOYMENT_ENV))!; + const indexerUrl = parseOption('--indexer-url', process.env.EVENT_CACHE_URL || DEFAULT_TESTNET_INDEXER_URL)!; + const rpcUrl = parseOption('--rpc-url', RPC_URL)!; + const expectedChainId = Number(parseOption('--chain-id', String(BASE_SEPOLIA_CHAIN_ID))); + const manifest = JSON.parse(await readFile(manifestPath, 'utf8')) as CampaignManifestV1; + const plan = await loadCampaignPlan(manifest, outputDirectory); + const environment = await loadCampaignEnvironment({ + mode: 'remote', + rpcUrl, + expectedChainId, + deploymentEnvPath, + mutationConfirmed: false, + }); + const { proposal, jsonPath, markdownPath } = await runRemoteCanaryPreflight({ + manifest, + plan, + environment, + chain: createCampaignChainAdapter(environment.rpcUrl), + indexerUrl, + outputDirectory, + pacingMs: Number(parseOption('--pacing-ms', String(DEFAULT_REMOTE_PACING_MS))), + concurrency: Number(parseOption('--concurrency', String(DEFAULT_REMOTE_CONCURRENCY))), + }); + console.log(formatCanaryProposalMarkdown(proposal)); + console.log(`Wrote ${jsonPath} and ${markdownPath}`); + if (!proposal.readyForAdamApproval) process.exitCode = 1; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main().catch((error) => { console.error(error instanceof Error ? error.message : error); process.exitCode = 1; }); +} diff --git a/fake-data-generation/package.json b/fake-data-generation/package.json index 68acf0e6..0955e837 100644 --- a/fake-data-generation/package.json +++ b/fake-data-generation/package.json @@ -20,6 +20,7 @@ "gen:campaign:plan": "tsx campaignPlanner.ts", "gen:campaign:execute": "tsx campaignExecute.ts", "gen:campaign:reconcile": "tsx campaignReconcile.ts", + "gen:campaign:canary-preflight": "tsx campaignCanary.ts", "gen:attesters": "tsx generateAttesters.ts", "gen:attestations": "tsx generateAttestations.ts", "gen:simulate": "tsx runSimulation.ts", diff --git a/fake-data-generation/test/campaignCanary.test.ts b/fake-data-generation/test/campaignCanary.test.ts new file mode 100644 index 00000000..4e3108e0 --- /dev/null +++ b/fake-data-generation/test/campaignCanary.test.ts @@ -0,0 +1,129 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { + BASE_SEPOLIA_CHAIN_ID, CAMPAIGN_CONTRACT_ENV_KEYS, loadCampaignEnvironment, preflightCampaignEnvironment, +} from '../campaignEnvironment.js'; +import { + REMOTE_CANARY_USER_COUNT, buildRemoteCanaryProposal, runRemoteCanaryPreflight, sliceCampaignPlanForCanary, +} from '../campaignCanary.js'; +import type { CampaignPlan, PlannedAction, PlannedUser } from '../campaignPlanner.js'; +import type { CampaignManifestV1 } from '../campaignSchema.js'; + +function user(id: string): PlannedUser { + return { + id, walletSlot: `wallet-${id}`, personaId: 'regular-supporter', roles: ['supporter'], + causeIds: ['open-source'], inactive: false, activityWeight: 1, fundingWeight: 1, + }; +} + +function action(partial: Partial & Pick): PlannedAction { + return { actorUserId: 'user-001', dependsOn: [], ...partial }; +} + +function plan(): CampaignPlan { + const users = Array.from({ length: 12 }, (_, index) => user(`user-${String(index + 1).padStart(3, '0')}`, index)); + return { + version: 'commonality-campaign-plan-v1', + campaignId: 'medium-realistic-v1', + deterministicSeed: 'seed', + manifestFingerprint: 'abc', + statements: [], + users, + projects: [{ id: 'project-001', title: 't', outcome: 'o', causeId: 'open-source', founderUserId: 'user-011', statementIds: [] }], + actions: [ + action({ id: 'action-00001', sequence: 1, type: 'publish-statement', actorUserId: null, statementId: 's1', dependsOn: [] }), + action({ id: 'action-00002', sequence: 2, type: 'create-cause', actorUserId: 'user-011', causeId: 'open-source', dependsOn: ['action-00001'] }), + action({ id: 'action-00003', sequence: 3, type: 'create-project', actorUserId: 'user-011', projectId: 'project-001', dependsOn: ['action-00002'] }), + action({ id: 'action-00004', sequence: 4, type: 'set-belief', actorUserId: 'user-001', statementId: 's1', belief: 'believe', dependsOn: ['action-00001'] }), + action({ id: 'action-00005', sequence: 5, type: 'fund-project', actorUserId: 'user-002', projectId: 'project-001', amount: 10, dependsOn: ['action-00003'] }), + action({ id: 'action-00006', sequence: 6, type: 'set-belief', actorUserId: 'user-012', statementId: 's1', belief: 'believe', dependsOn: ['action-00001'] }), + ], + estimate: { + writesByType: {} as CampaignPlan['estimate']['writesByType'], + totalWrites: 6, + estimatedGasByType: {} as CampaignPlan['estimate']['estimatedGasByType'], + estimatedTotalGas: 0, + assumptions: { gasUnitsPerWrite: {} as CampaignPlan['estimate']['assumptions']['gasUnitsPerWrite'], paymentTokenBaseUnit: 100 }, + estimatedPaymentTokenUnits: 0, + }, + }; +} + +function manifest(): CampaignManifestV1 { + return { + campaign: { id: 'medium-realistic-v1', syntheticDataLabel: 'SYNTHETIC TESTNET CAMPAIGN — NOT REAL USERS OR ADOPTION' }, + artifactLayout: { walletSecrets: '../secrets/medium-realistic-v1.wallets.json' }, + } as CampaignManifestV1; +} + +async function withEnv(run: (deploymentEnvPath: string) => Promise): Promise { + const directory = await mkdtemp(path.join(tmpdir(), 'campaign-canary-')); + const deploymentEnvPath = path.join(directory, 'deployment.env'); + const lines = Object.values(CAMPAIGN_CONTRACT_ENV_KEYS).map((key, index) => `${key}=0x${String(index + 1).padStart(40, '0')}`); + await writeFile(deploymentEnvPath, `${lines.join('\n')}\n`); + try { await run(deploymentEnvPath); } finally { await rm(directory, { recursive: true, force: true }); } +} + +test('canary slice keeps the first 10 users plus dependency actors', () => { + const slice = sliceCampaignPlanForCanary(plan(), 10); + assert.equal(slice.userCount, REMOTE_CANARY_USER_COUNT); + assert.equal(slice.users.length, 10); + assert.ok(slice.extraActors.some((actor) => actor.id === 'user-011')); + assert.ok(slice.actions.some((item) => item.id === 'action-00005')); + assert.equal(slice.actions.some((item) => item.id === 'action-00006'), false); + assert.ok(slice.actions.some((item) => item.type === 'publish-statement')); + assert.ok(slice.projects.some((project) => project.id === 'project-001')); +}); + +test('preflight proposal is read-only and fail-closed on indexer lag', async () => withEnv(async (deploymentEnvPath) => { + const environment = await loadCampaignEnvironment({ + mode: 'remote', rpcUrl: 'https://sepolia.base.org', expectedChainId: BASE_SEPOLIA_CHAIN_ID, deploymentEnvPath, + }); + const chainPreflight = await preflightCampaignEnvironment(environment, { + getChainId: async () => BASE_SEPOLIA_CHAIN_ID, getBytecode: async () => '0x01', + }); + const proposal = buildRemoteCanaryProposal({ + plan: plan(), manifest: manifest(), environment, chainPreflight, + indexerUrl: 'https://commonality-indexer.onrender.com', indexerLagBlocks: 0n, + }); + assert.equal(proposal.mutatesChain, false); + assert.equal(proposal.phase, 'remote-canary-10'); + assert.equal(proposal.userCount, 10); + assert.ok(Number(proposal.nativeWeiNeeded) > 0); + assert.ok(proposal.gates.some((gate) => gate.id === 'budget-and-window' && gate.status === 'needs-adam')); + assert.equal(proposal.readyForAdamApproval, true); + assert.equal(proposal.secrets.hardhatKeys, 'forbidden on remote'); + const lagFail = buildRemoteCanaryProposal({ + plan: plan(), manifest: manifest(), environment, chainPreflight, + indexerUrl: 'https://commonality-indexer.onrender.com', indexerLagBlocks: 10_000n, + }); + assert.equal(lagFail.gates.find((gate) => gate.id === 'indexer-lag')?.status, 'fail'); + assert.equal(lagFail.readyForAdamApproval, false); +})); + +test('runRemoteCanaryPreflight writes artifacts and never confirms mutation', async () => withEnv(async (deploymentEnvPath) => { + const outputDirectory = await mkdtemp(path.join(tmpdir(), 'campaign-canary-out-')); + try { + const environment = await loadCampaignEnvironment({ + mode: 'remote', rpcUrl: 'https://sepolia.base.org', expectedChainId: BASE_SEPOLIA_CHAIN_ID, deploymentEnvPath, + }); + const { proposal, jsonPath, markdownPath } = await runRemoteCanaryPreflight({ + manifest: manifest(), plan: plan(), environment, + chain: { getChainId: async () => BASE_SEPOLIA_CHAIN_ID, getBytecode: async () => '0x01' }, + indexerUrl: 'https://example.invalid', + outputDirectory, + probeIndexer: async () => ({ chainHead: 100n, indexerHead: 100n }), + }); + assert.equal(environment.mutationConfirmed, false); + assert.equal(proposal.indexerLagBlocks, '0'); + const written = JSON.parse(await readFile(jsonPath, 'utf8')) as { mutatesChain: boolean }; + assert.equal(written.mutatesChain, false); + const markdown = await readFile(markdownPath, 'utf8'); + assert.match(markdown, /does not mutate/); + } finally { + await rm(outputDirectory, { recursive: true, force: true }); + } +})); From d99d16bde5722b842314b9bda8b1958242ccf0ba Mon Sep 17 00:00:00 2001 From: Adam Spitz Date: Fri, 11 Sep 2026 17:36:07 -0400 Subject: [PATCH 19/19] Temporarily disable CreatorAssuranceVeto indexer (not yet deployed to Base Sepolia) The CreatorAssuranceVeto contract was extracted on Sep 10 but hasn't been redeployed to Base Sepolia yet. The current deployed factory doesn't have the contentVeto() function. This allows the indexer to start while we wait for contract redeployment. --- indexer/ponder.config.ts | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/indexer/ponder.config.ts b/indexer/ponder.config.ts index a0a639e4..341c1df1 100644 --- a/indexer/ponder.config.ts +++ b/indexer/ponder.config.ts @@ -40,7 +40,7 @@ import { ContentRegistryAbi } from "./abis/ContentRegistryAbi"; import { BeneficiaryRegistryAbi } from "./abis/BeneficiaryRegistryAbi"; import { BeneficiaryEscrowAbi } from "./abis/BeneficiaryEscrowAbi"; import { CreatorAssuranceContractFactoryAbi } from "./abis/CreatorAssuranceContractFactoryAbi"; -import { CreatorAssuranceVetoAbi } from "./abis/CreatorAssuranceVetoAbi"; +// import { CreatorAssuranceVetoAbi } from "./abis/CreatorAssuranceVetoAbi"; // TODO: Uncomment after redeploying contracts import { ProspectiveContentRoundFactoryAbi } from "./abis/ProspectiveContentRoundFactoryAbi"; import { MaterializedContentTokensAbi } from "./abis/MaterializedContentTokensAbi"; @@ -216,7 +216,7 @@ const CONTENT_REGISTRY_DEPLOYMENTS = getDeployments("ContentRegistry", "CONTENT_ const BENEFICIARY_REGISTRY_DEPLOYMENTS = getDeployments("BeneficiaryRegistry", "BENEFICIARY_REGISTRY_ADDRESS", CONTENT_FUNDING_START_BLOCK); const BENEFICIARY_ESCROW_DEPLOYMENTS = getDeployments("BeneficiaryEscrow", "BENEFICIARY_ESCROW_ADDRESS", CONTENT_FUNDING_START_BLOCK); const CREATOR_CONTRACT_FACTORY_DEPLOYMENTS = getDeployments("CreatorAssuranceContractFactory", "CREATOR_CONTRACT_FACTORY_ADDRESS", CONTENT_FUNDING_START_BLOCK); -const CREATOR_ASSURANCE_VETO_DEPLOYMENTS = getDeployments("CreatorAssuranceVeto", "CREATOR_ASSURANCE_VETO_ADDRESS", CONTENT_FUNDING_START_BLOCK); +// const CREATOR_ASSURANCE_VETO_DEPLOYMENTS = getDeployments("CreatorAssuranceVeto", "CREATOR_ASSURANCE_VETO_ADDRESS", CONTENT_FUNDING_START_BLOCK); // TODO: Uncomment after redeploying contracts const PROSPECTIVE_FACTORY_DEPLOYMENTS = getDeployments("ProspectiveContentRoundFactory", "PROSPECTIVE_CONTENT_ROUND_FACTORY_ADDRESS", CONTENT_FUNDING_START_BLOCK); const ETH_GET_LOGS_BLOCK_RANGE = process.env.PONDER_ETH_GET_LOGS_BLOCK_RANGE @@ -421,11 +421,13 @@ const contracts = { ...deploymentConfig(CREATOR_CONTRACT_FACTORY_DEPLOYMENTS, CONTENT_FUNDING_START_BLOCK), }, - CreatorAssuranceVeto: { - abi: CreatorAssuranceVetoAbi, - chain: INDEXER_CHAIN, - ...deploymentConfig(CREATOR_ASSURANCE_VETO_DEPLOYMENTS, CONTENT_FUNDING_START_BLOCK), - }, + // TODO: CreatorAssuranceVeto not yet deployed to Base Sepolia (extracted Sep 10) + // Uncomment after redeploying contracts + // CreatorAssuranceVeto: { + // abi: CreatorAssuranceVetoAbi, + // chain: INDEXER_CHAIN, + // ...deploymentConfig(CREATOR_ASSURANCE_VETO_DEPLOYMENTS, CONTENT_FUNDING_START_BLOCK), + // }, ProspectiveContentRoundFactory: { abi: ProspectiveContentRoundFactoryAbi,