FE-1990: Add a reusable way to run flows end to end

[kostandinang]

🌟 Purpose

Flow actions only have unit tests. This adds a way to run flows end to end on the local stack: a scenario is one or more flow runs, each a flow definition, its inputs and the expected outputs, optionally with entities to create first. run-flow-scenario runs them through Temporal, checks every step and prints a report for the PR.

yarn workspace @apps/hash-ai-worker-ts run-flow-scenario scripts/flow-scenarios/scenarios/query-entities.scenario.ts

It comes with three scenarios for what's on main:

• Array actions: Filter List, Order By, Take First, Aggregate, Find and If on a list of numbers.
• Query entities: seeds three Docs, finds them by the run's tag, then filters, sorts and aggregates them on properties read from the graph. Query entities (FE-1983: Add a Query entities flow action #10050) had only been tested against a mocked graph.
• forEach: the engine itself: steps run once per item, results are collected in item order, and a forEach over an empty list completes with nothing collected.

🔍 How it works

scenario.ts ──▶ run-flow-scenario
                 │ 1. validate each run's flow and expected step ids   (--dry-run stops here)
                 │ 2. check Temporal and a worker from this checkout   ──✗──▶ exit 2
                 │ 3. seed entities, all checked against their types first
                 │ for each run, in order:
                 │   4. upsert its Flow Definition in the user's web
                 │   5. start runFlow, like startFlow does, and wait
                 ▼
          read run ─▶ check expectations ─┬─ all met ─▶ next run
                                          └─ any ✗  ─▶ stop
                 ▼
          report.md ─▶ exit 0, or 1 + flow-run-<id>.json

Everything is in apps/hash-ai-worker-ts/scripts/flow-scenarios/, and its README covers starting the stack and writing a scenario.

🧐 To review

• Definitions go in the user's web. Migrations only seed HASH's own flows into the h web. An entity uuid can only be used in one web, so the definition's uuid is derived from the scenario's flowDefinitionId and the user.
• Run tag. Seeded entities stay in the web, so a query would also find those from earlier runs. A value unique to each run of the scenario is passed to seed and inputs to mark the entities and query for exactly those.
• Why the runner seeds. On main, the only action that creates entities is persistEntity(ies), which may update a similar existing entity (matched by OpenAI embeddings) instead of creating one. Seeding through the graph keeps the scenario exact.
• Why several runs. A step only waits for the steps whose outputs it takes, so a Query entities step can't be ordered after the steps that create entities in the same flow. Creating and then querying takes two runs.

🚫 Out of scope

Scenarios for actions not yet on main come with those actions: an entity round trip is ready to go with #10060. CI comes after FE-1904.

🛡 Tests

On a local stack (the ai worker needs a real ANTHROPIC_API_KEY in .env.local, since every run asks a model to name itself):

• Array actions meets 20 of 20 expectations, Query entities 31 of 31 and forEach 38 of 38 (reports below). Run again, Query entities finds only that run's 3 Docs.
• The entity round trip, on top of FE-1985: Add Create entity and Update entity flow actions #10060, meets 32 of 32 over two runs.
• Failure paths: a wrong expected value exits 1 and stops later runs. A seed value that doesn't fit its type, an expected step id that can't match a step run, no worker, and a worker from another checkout each exit 2 with a one-line reason, before anything is created.
• tsc --noEmit and ESLint pass.

Array actions run report

Flow scenario: Array actions on a list of numbers

Chains Filter List, Order By, Take First and Aggregate, branches on the total with If, and checks the untaken branch is skipped.

Local stack at b212d219fd, started through Temporal. Run tag ed46c55f-d26b-4574-b1e0-198675487623.

Run 1 of 1: Array actions run

873ffc25-39d0-4a3b-b24b-268dea35b845: COMPLETED

Step	Action	Status	Outputs
passing	filterList	COMPLETED	matching: 4 × Number [72,90,50,99] nonMatching: 3 × Number [45,8,30]
passingCount	aggregate	COMPLETED	result: Number 4
ranked	orderBy	COMPLETED	items: 4 × Number [99,90,72,50]
top	takeFirst	COMPLETED	items: 2 × Number [99,90]
total	aggregate	COMPLETED	result: Number 189
isBig	if	COMPLETED	true: Number 189
whenSmall	takeFirst	SKIPPED	–
firstLow	find	COMPLETED	match: Number 8

Expectations: 20 of 20 met

• ✓ flow is COMPLETED
• ✓ passing is COMPLETED
• ✓ passing.matching
• ✓ passing.nonMatching
• ✓ passingCount is COMPLETED
• ✓ passingCount.result
• ✓ ranked is COMPLETED
• ✓ ranked.items
• ✓ top is COMPLETED
• ✓ top.items
• ✓ total is COMPLETED
• ✓ total.result
• ✓ isBig is COMPLETED
• ✓ isBig.true
• ✓ isBig.false not produced
• ✓ whenSmall is SKIPPED
• ✓ firstLow is COMPLETED
• ✓ firstLow.match
• ✓ flow output total
• ✓ flow output firstLow

Expectations: 20 of 20 met

Query entities run report

Flow scenario: Query entities on seeded Docs

Finds seeded Docs by the run's tag with Query entities, filters, sorts and aggregates them on properties read from the graph, and checks the graph's own sorting and limit.

Local stack at b212d219fd, started through Temporal. Run tag d01bc29b-15d0-4e22-9dd0-8da2285e4a66, 3 entities seeded.

Run 1 of 1: Query tagged docs

fd4d37ed-6d9a-4fef-9e69-52397f0f8e0f: COMPLETED

Step	Action	Status	Outputs
found	queryEntities	COMPLETED	entities: 3 × EntityId ["cbf261a6-8a16-4436-ba51-2dd64e67aeddc019806b-9474-4e05-b147-22f7d8615332","cbf261a6-8a16-4436-ba51-2dd64e67aedd40396128-b155-421d-8786-…
foundCount	aggregate	COMPLETED	result: Number 3
totalPages	aggregate	COMPLETED	result: Number 168
long	filterList	COMPLETED	matching: 2 × EntityId ["cbf261a6-8a16-4436-ba51-2dd64e67aeddc019806b-9474-4e05-b147-22f7d8615332","cbf261a6-8a16-4436-ba51-2dd64e67aedd898ac73e-049c-4fda-a4a2-… nonMatching: 1 × EntityId ["cbf261a6-8a16-4436-ba51-2dd64e67aedd~40396128-b155-421d-8786-89b0f50e47d5"]
longCount	aggregate	COMPLETED	result: Number 2
earliestLongYear	aggregate	COMPLETED	result: Number 2021
byYear	orderBy	COMPLETED	items: 2 × EntityId ["cbf261a6-8a16-4436-ba51-2dd64e67aedd898ac73e-049c-4fda-a4a2-d6daf4a44d9c","cbf261a6-8a16-4436-ba51-2dd64e67aeddc019806b-9474-4e05-b147-…
earliestLong	takeFirst	COMPLETED	items: 1 × EntityId ["cbf261a6-8a16-4436-ba51-2dd64e67aedd~898ac73e-049c-4fda-a4a2-d6daf4a44d9c"]
earliestLongPages	aggregate	COMPLETED	result: Number 40
short	find	COMPLETED	match: EntityId "cbf261a6-8a16-4436-ba51-2dd64e67aedd~40396128-b155-421d-8786-89b0f50e47d5"
shortPages	aggregate	COMPLETED	result: Number 8
huge	find	COMPLETED	–
whenHuge	takeFirst	SKIPPED	–
firstTwoByTitle	queryEntities	COMPLETED	entities: 2 × EntityId ["cbf261a6-8a16-4436-ba51-2dd64e67aeddc019806b-9474-4e05-b147-22f7d8615332","cbf261a6-8a16-4436-ba51-2dd64e67aedd898ac73e-049c-4fda-a4a2-…
firstTwoByTitlePages	aggregate	COMPLETED	result: Number 160

Expectations: 31 of 31 met

• ✓ flow is COMPLETED
• ✓ found is COMPLETED
• ✓ foundCount is COMPLETED
• ✓ foundCount.result
• ✓ totalPages is COMPLETED
• ✓ totalPages.result
• ✓ long is COMPLETED
• ✓ longCount is COMPLETED
• ✓ longCount.result
• ✓ earliestLongYear is COMPLETED
• ✓ earliestLongYear.result
• ✓ byYear is COMPLETED
• ✓ earliestLong is COMPLETED
• ✓ earliestLongPages is COMPLETED
• ✓ earliestLongPages.result
• ✓ short is COMPLETED
• ✓ shortPages is COMPLETED
• ✓ shortPages.result
• ✓ huge is COMPLETED
• ✓ huge.match not produced
• ✓ whenHuge is SKIPPED
• ✓ firstTwoByTitle is COMPLETED
• ✓ firstTwoByTitlePages is COMPLETED
• ✓ firstTwoByTitlePages.result
• ✓ flow output foundCount
• ✓ flow output totalPages
• ✓ flow output longCount
• ✓ flow output earliestLongYear
• ✓ flow output earliestLongPages
• ✓ flow output shortPages
• ✓ flow output firstTwoByTitlePages

Expectations: 31 of 31 met

forEach run report

Flow scenario: forEach

Runs steps once per item in parallel, with one item feeding two steps and a step that only some items run, collects the results in item order, and completes a forEach over an empty list with nothing collected.

Local stack at b212d219fd, started through Temporal. Run tag beb6ed95-2b01-4e19-8b41-0395b5fb4375.

Run 1 of 1: forEach run

b7750508-4020-4668-abe1-a870d75032b9: COMPLETED

Step	Action	Status	Outputs
passed~0	if	COMPLETED	true: Number 72
passed~1	if	COMPLETED	false: Number 8
passed~2	if	COMPLETED	true: Number 99
passed~3	if	COMPLETED	false: Number 30
keepPassing~0	takeFirst	COMPLETED	items: 1 × Number [72]
keepPassing~1	takeFirst	SKIPPED	–
keepPassing~2	takeFirst	COMPLETED	items: 1 × Number [99]
keepPassing~3	takeFirst	SKIPPED	–
wrap~0	takeFirst	COMPLETED	items: 1 × Number [72]
wrap~1	takeFirst	COMPLETED	items: 1 × Number [8]
wrap~2	takeFirst	COMPLETED	items: 1 × Number [99]
wrap~3	takeFirst	COMPLETED	items: 1 × Number [30]
total	aggregate	COMPLETED	result: Number 209
huge	filterList	COMPLETED	matching: 0 × Number [] nonMatching: 4 × Number [72,8,99,30]
wrapHuge~<index>	takeFirst	NOT RUN	for no item
hugeCount	aggregate	COMPLETED	result: Number 0

Expectations: 38 of 38 met

• ✓ flow is COMPLETED
• ✓ passed~0 is COMPLETED
• ✓ passed~0.true
• ✓ passed~0.false not produced
• ✓ passed~1 is COMPLETED
• ✓ passed~1.false
• ✓ passed~1.true not produced
• ✓ passed~2 is COMPLETED
• ✓ passed~2.true
• ✓ passed~2.false not produced
• ✓ passed~3 is COMPLETED
• ✓ passed~3.false
• ✓ passed~3.true not produced
• ✓ keepPassing~0 is COMPLETED
• ✓ keepPassing~0.items
• ✓ keepPassing~1 is SKIPPED
• ✓ keepPassing~2 is COMPLETED
• ✓ keepPassing~2.items
• ✓ keepPassing~3 is SKIPPED
• ✓ wrap~0 is COMPLETED
• ✓ wrap~0.items
• ✓ wrap~1 is COMPLETED
• ✓ wrap~1.items
• ✓ wrap~2 is COMPLETED
• ✓ wrap~2.items
• ✓ wrap~3 is COMPLETED
• ✓ wrap~3.items
• ✓ total is COMPLETED
• ✓ total.result
• ✓ huge is COMPLETED
• ✓ huge.matching
• ✓ huge.nonMatching
• ✓ hugeCount is COMPLETED
• ✓ hugeCount.result
• ✓ flow output collected
• ✓ flow output total
• ✓ flow output collectedFromNone
• ✓ flow output hugeCount

Expectations: 38 of 38 met

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Oct 19, 2026 4:45pm UTC

3 Skipped Deployments

Project	Deployment	Actions	Updated
hashdotdesign-tokens	Ignored	Preview	Oct 19, 2026 4:45pm UTC
petrinaut	Skipped		Oct 19, 2026 4:45pm UTC
petrinaut-docs	Skipped		Oct 19, 2026 4:45pm UTC

[codecov]

Codecov Report

❌ Patch coverage is 0% with 338 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.02%. Comparing base (d3c0347) to head (5976485).

Files with missing lines	Patch %	Lines
...ker-ts/scripts/flow-scenarios/run-flow-scenario.ts	0.00%	280 Missing ⚠️
...low-scenarios/scenarios/query-entities.scenario.ts	0.00%	30 Missing ⚠️
...ipts/flow-scenarios/scenarios/for-each.scenario.ts	0.00%	15 Missing ⚠️
...flow-scenarios/scenarios/array-actions.scenario.ts	0.00%	12 Missing ⚠️
...sh-ai-worker-ts/scripts/flow-scenarios/scenario.ts	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10052      +/-   ##
==========================================
- Coverage   68.13%   68.02%   -0.12%     
==========================================
  Files        1995     2000       +5     
  Lines      205556   205894     +338     
  Branches     8708     8782      +74     
==========================================
  Hits       140053   140053              
- Misses      63838    64176     +338     
  Partials     1665     1665              

Flag	Coverage Δ
apps.hash-ai-worker-ts	6.86% <0.00%> (-0.42%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.