Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## Unreleased

- Stability report (S8), 2026-11-15 (K408). `compareRuns` compares two grouping runs of one kit, matching branch labels by overlap so a renumbering isn't counted, and `assertStable` stops when more than 5% of the people placed in both runs change line (side and branch). Deeper levels, groups, newly placed and no longer placed are reported. Tested on made-up runs and a planted family; on real data it rated the original method's run against OneFamily's as stable (1.9%). The app doesn't call it yet, since it keeps one run per kit.

- Genetic map download, 2026-11-15 (K653, owner's N124 A). A Genetic map card on the Kits screen offers the HapMap-derived GRCh37 map (about 24 MB) from the Beagle site at the University of Washington. It says where the map comes from and that no license is stated, downloads only after a yes, checks the pinned size and SHA-256, and can remove the map again. Comparisons use it when both raw files are on GRCh37; the demo always uses the approximate rule. Tested with made-up archives, and once by hand with the real archive (chromosome 1 comes out at 286 cM). The card hasn't been clicked in the built app.

- The cross-kit check (S7) is enforced, 2026-11-15 (K655). Phasing a kit with a parent now stops unless the two share parent-child cM, and rebuilding someone also needs the children to share full-sibling cM. `fittingRelationships` and `RELATIONSHIP_WORDS` say which relationships a cM fits, and the Kits screen lists them for each pair compared on the Mac. Tested on simulated kits; the screen text hasn't been seen in the built app.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ DNA-first genetic genealogy for Mac. OneFamily groups your DNA matches before yo
- OneFamily for Mac, v0: runs on a made-up demo family. It's been built and opened locally, not released. It carries its own copy of Node, so nothing extra needs installing (ADR 0007).
- the Chrome extension the app collects with (`apps/extension`, ADR 0009). It stores nothing, talks to the app only through Chrome's native messaging, and asks for your OK in the app. It only works with a local mock site: every DNA service's terms say no to automated page visits. It's been tested with Vitest against the mock site, and hasn't been loaded into Chrome yet.

275 tests pass locally. The raw layouts haven't been checked against a real downloaded file. The register's verdicts are a careful reading of public pages, not legal advice, and a legal review is still to come.
282 tests pass locally. The raw layouts haven't been checked against a real downloaded file. The register's verdicts are a careful reading of public pages, not legal advice, and a legal review is still to come.

## What it will do

Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/0003-safeguards.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Each safeguard is enforced in code and covered by tests. A promise that's only w
| S5 | Count reconciliation | Unique IDs; previous export + new − gone = this export; the row count equals the count the service showed, when recorded |
| S6 | Spot checks against the source | A sample of matches compared with the service's own pages, where the terms register allows it |
| S7 | Genetic sanity across kits | Family kits share cM in range for their DNA relationship; side assignments agree across kits. Built in `sanity.ts` (2026-11-15): out-of-range pairs, missing close relatives and more than 5% of shared matches on the wrong side stop the run; relationships with no published range are reported as not checked. Enforced since 2026-11-15 (K655): `phaseWithParent` and `rebuildFromDescendants` measure the raw kits first and stop unless the parent and child share parent-child cM and the children share full-sibling cM; a relationship with no range can't be relied on there. The Kits screen lists what each measured pair fits. Any later analysis that combines kits must call `assertSane` first |
| S8 | Stability report | A run stops for review if more than a few percent of placed people change lines |
| S8 | Stability report | A run stops for review if more than a few percent of placed people change lines. Built 2026-11-15 (K408): `compareRuns` and `assertStable` in `@extantape/onefamily-analysis`. Labels are matched by overlap, so renumbering isn't change. A line is the side-split path to two levels; more than 5% of the people placed in both runs changing line stops the run. Deeper levels and groups are reported, not judged. Under 50 people placed in both runs gets no verdict, and says so. The app keeps one run per kit so far, so nothing in it calls this yet; a check that replaces a kit's export must |
| S9 | Claims checked against data | Every match cited in a theory is checked against the stored observations |
| S10 | People decide conclusions | A theory's status changes only with the user's approval, which is logged |
| S11 | Backups and rebuild check | The cache is backed up before imports; a full rebuild from the library must equal the live cache |
Expand Down
2 changes: 1 addition & 1 deletion docs/test-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Kept by this repo's owning session (Genealogy), per `~/Code/CLAUDE.md`, "Test se
- **Theory chart, both candidate fathers (K337):** 2026-11-15, `apps/desktop/test/logic.test.ts` checks that a theory document drawn with OnePedigree's `uncertainParents: 'branches'` shows both the recorded and the candidate father, each joined by an uncertain line. The screen's radio switch itself is covered only by the typecheck; it hasn't been clicked in the built app.
- **Bundled Node (K338):** 2026-11-15, checked by hand: `tauri build --bundles app`, then the app's own `Contents/MacOS/node` ran `host.mjs` under `env -i` (no PATH) and answered a `status` request. That run found the chunk and `node:sqlite` faults, now fixed. Not automated: CI doesn't build the Mac app yet, and a unit test of the download script would only re-check the pinned hash. Worth adding a CI step that runs the built helper once the Mac build moves to CI.
- **Genetic map download (K653):** 2026-11-15, `apps/desktop/test/genetic-map.test.ts` (5 tests): stored and deflated zip entries; install writes a note naming the source and converts positions; a wrong size, a wrong checksum or a missing chromosome writes nothing; the host refuses without a yes and calls the downloader only once after it; the demo keeps the approximate rule. The real archive was checked by hand once, not in CI, to keep CI off the network. The card hasn't been clicked in the built app.
- **S8, the stability report between runs:** not built, since there's only one run so far. Filed as a task.
- **S8, the stability report between runs:** built 2026-11-15 (K408), covered by `packages/analysis/test/stability.test.ts` (7 tests): a run against itself, renumbered branches, 10 of 200 moved passes and 11 stops, deeper changes reported but not judged at the default depth, newly placed and no longer placed, too few to judge, different kits refused, and a planted rerun with matches removed. On real data it compared the original method's run of the first finished kit with OneFamily's: 1.9% changed line, stable. "Too few" doesn't stop a run; it's reported, like S7's side check.
- **S9, claims checked against the data:** theories aren't stored as data yet. Filed as a task.
- **S10, the owner approves theory status changes:** no theory editing exists yet. It's part of S9's task.
- **Native-only behaviour:** the Mac notification, the menu item, the folder picker, the real Keychain, and the extension inside Chrome. These need the running app or the owner's Chrome, so they're listed for a by-hand test before the first release.
Expand Down
1 change: 1 addition & 0 deletions packages/analysis/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ export { nestedSplit, twoWaySplit, type NestedSplit, type SplitOptions, type Spl
export { propagateFromSeeds, UNPLACED, type Placement, type PropagateOptions } from './propagate.js'
export { adjustedRandIndex } from './ari.js'
export { clusterKit, ENGINE, type ClusterOptions, type ClusterRun, type GroupedMatch } from './cluster.js'
export { assertStable, compareRuns, StabilityStop, type LevelReport, type StabilityOptions, type StabilityReport } from './stability.js'
118 changes: 118 additions & 0 deletions packages/analysis/src/stability.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
/**
* Safeguard S8, the stability report between runs (ADR 0003). A new run of the same kit, after a check or a
* full rerun, should leave most people where they were. If more than a few percent of the people placed in
* both runs change line, the run stops for review: something in the data or the method moved, and every
* finding built on the old lines needs another look.
*
* Labels aren't comparable as they stand: groups are numbered by size and side splits are numbered within
* each side, so a renumbering alone would look like everyone moving. Each level's labels in the old run are
* matched one to one with the new run's labels they overlap most, and a person changed line only if their
* new label isn't the match of their old one.
*
* Lines are compared to `depth` levels of the side-split path (default 2: side, then branch). Deeper levels
* are reported too, as notices: third-level splits are often borderline, and the method's own reruns
* differed there. Groups are reported as a notice with their adjusted Rand index.
*/
import type { ClusterRun } from './cluster.js'
import { adjustedRandIndex } from './ari.js'

export type StabilityOptions = {
/** Path levels that define a line. Default 2. */
depth?: number
/** More changes than this share of the people placed in both runs stops the run. Default 0.05. */
maxChanged?: number
/** Fewer people placed in both runs than this can't support a verdict. Default 50. */
minCompared?: number
}

export type LevelReport = { depth: number; compared: number; changed: number; share: number; ari: number }

export type StabilityReport = {
kit: string
verdict: 'stable' | 'stop' | 'too-few'
depth: number
maxChanged: number
/** Placed in both runs at `depth`. */
compared: number
changed: { match: string; before: string; after: string }[]
share: number
/** Placed before but not now, and the other way round. */
noLongerPlaced: string[]
newlyPlaced: string[]
levels: LevelReport[]
groupsAri: number
message: string
}

export class StabilityStop extends Error {
constructor(public readonly report: StabilityReport) {
super(`stopped by the stability check (S8): ${report.message}`)
this.name = 'StabilityStop'
}
}

const cut = (path: string, depth: number) => path.split('.').slice(0, depth).join('.')
/** A path counts at a depth only if it goes that deep and has no unsure step ("?"). */
const placedAt = (path: string | null, depth: number): string | null => {
if (!path) return null
const parts = path.split('.')
if (parts.length < depth || parts.slice(0, depth).some((p) => p.includes('?'))) return null
return cut(path, depth)
}

/** One-to-one matching of old labels to new ones by overlap, largest overlaps first. Ties go to the label sorted first. */
function matchLabels(pairs: readonly [string, string][]): Map<string, string> {
const counts = new Map<string, number>()
for (const [a, b] of pairs) counts.set(`${a}\u0000${b}`, (counts.get(`${a}\u0000${b}`) ?? 0) + 1)
const ranked = [...counts].map(([k, n]) => { const [a, b] = k.split('\u0000') as [string, string]; return { a, b, n } })
.sort((x, y) => y.n - x.n || (x.a < y.a ? -1 : x.a > y.a ? 1 : x.b < y.b ? -1 : x.b > y.b ? 1 : 0))
const map = new Map<string, string>()
const used = new Set<string>()
for (const { a, b } of ranked) if (!map.has(a) && !used.has(b)) { map.set(a, b); used.add(b) }
return map
}

function level(before: Map<string, string | null>, after: Map<string, string | null>, depth: number) {
const pairs: [string, string, string][] = []
for (const [m, p] of before) {
const a = placedAt(p, depth)
const b = placedAt(after.get(m) ?? null, depth)
if (a && b) pairs.push([m, a, b])
}
const map = matchLabels(pairs.map(([, a, b]) => [a, b]))
const changed = pairs.filter(([, a, b]) => map.get(a) !== b).map(([match, a, b]) => ({ match, before: a, after: b }))
const ari = adjustedRandIndex(new Map(pairs.map(([m, a]) => [m, a])), new Map(pairs.map(([m, , b]) => [m, b])))
return { compared: pairs.length, changed, ari }
}

export function compareRuns(before: ClusterRun, after: ClusterRun, opts: StabilityOptions = {}): StabilityReport {
const { depth = 2, maxChanged = 0.05, minCompared = 50 } = opts
if (before.kit !== after.kit) throw new Error(`the two runs are for different kits (${before.kit}, ${after.kit})`)
const b = new Map(before.matches.map((m) => [m.match, m.path]))
const a = new Map(after.matches.map((m) => [m.match, m.path]))
const main = level(b, a, depth)
const maxDepth = Math.max(depth, ...[...b.values(), ...a.values()].map((p) => (p ? p.split('.').length : 0)))
const levels: LevelReport[] = []
for (let d = 1; d <= maxDepth; d++) {
const l = d === depth ? main : level(b, a, d)
levels.push({ depth: d, compared: l.compared, changed: l.changed.length, share: l.compared ? l.changed.length / l.compared : 0, ari: l.ari })
}
const noLongerPlaced = [...b].filter(([m, p]) => placedAt(p, depth) && !placedAt(a.get(m) ?? null, depth)).map(([m]) => m).sort()
const newlyPlaced = [...a].filter(([m, p]) => placedAt(p, depth) && !placedAt(b.get(m) ?? null, depth)).map(([m]) => m).sort()
const bg = new Map(before.matches.map((m) => [m.match, m.group]))
const ag = new Map(after.matches.map((m) => [m.match, m.group]))
const groupsAri = adjustedRandIndex(bg, ag)
const share = main.compared ? main.changed.length / main.compared : 0
const verdict = main.compared < minCompared ? 'too-few' : share > maxChanged ? 'stop' : 'stable'
const pct = (x: number) => `${(x * 100).toFixed(1)}%`
const message =
verdict === 'too-few'
? `only ${main.compared} people are placed in both runs, fewer than ${minCompared}, so stability isn't judged`
: `${main.changed.length} of ${main.compared} people placed in both runs changed line (${pct(share)}, limit ${pct(maxChanged)}, lines to ${depth} levels)`
return { kit: before.kit, verdict, depth, maxChanged, compared: main.compared, changed: main.changed, share, noLongerPlaced, newlyPlaced, levels, groupsAri, message }
}

/** Stops unless the new run is stable against the old one. "Too few" doesn't stop, but says so. */
export function assertStable(report: StabilityReport): void {
if (report.verdict === 'stop') throw new StabilityStop(report)
}
74 changes: 74 additions & 0 deletions packages/analysis/test/stability.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
import { describe, expect, it } from 'vitest'
import { assertStable, clusterKit, compareRuns, SeededRandom, StabilityStop, type ClusterRun } from '../src/index.js'
import { plantedKit } from './planted.js'

/** A made-up run: `paths[i]` is match i's side-split path. */
const run = (paths: (string | null)[], kit = 'K'): ClusterRun => ({
engine: { name: '@extantape/onefamily-analysis', version: '0.1.0' },
kit,
inputs: [],
settings: {} as ClusterRun['settings'],
matches: paths.map((path, i) => ({ match: `m${i}`, group: `G${path ?? 'x'}`, stability: 1, path })),
excluded: [],
splits: [],
})
const base = Array.from({ length: 200 }, (_, i) => ['M.1.1', 'M.1.2', 'M.2', 'P.1', 'P.2'][i % 5]!)

describe('the stability report between runs (S8)', () => {
it('finds nothing changed between a run and itself', () => {
const r = compareRuns(run(base), run(base))
expect(r).toMatchObject({ verdict: 'stable', compared: 200, share: 0, changed: [] })
expect(r.groupsAri).toBe(1)
})

it('treats a renumbering as no change: branches are matched by who is in them', () => {
const swapped = base.map((p) => p.replace(/^M\.1/, 'M.X').replace(/^M\.2/, 'M.1').replace(/^M\.X/, 'M.2'))
expect(compareRuns(run(base), run(swapped)).changed).toEqual([])
})

it('stops when more than 5% of the people placed in both runs change line, and passes at 5% or less', () => {
const move = (n: number) => base.map((p, i) => (i < n * 5 && p === 'P.1' ? 'P.2' : p))
const few = compareRuns(run(base), run(move(10)))
expect(few.changed).toHaveLength(10)
expect(few.verdict).toBe('stable')
expect(() => assertStable(few)).not.toThrow()
const many = compareRuns(run(base), run(move(11)))
expect(many.verdict).toBe('stop')
expect(many.changed[0]).toEqual({ match: 'm3', before: 'P.1', after: 'P.2' })
expect(() => assertStable(many)).toThrow(StabilityStop)
expect(() => assertStable(many)).toThrow(/11 of 200 .*5\.5%/)
})

it('judges lines to two levels by default and reports deeper changes without stopping', () => {
const deep = base.map((p, i) => (p === 'M.1.1' && i < 100 ? 'M.1.2' : p))
const r = compareRuns(run(base), run(deep))
expect(r.verdict).toBe('stable')
expect(r.levels.find((l) => l.depth === 3)!.changed).toBe(20)
expect(compareRuns(run(base), run(deep), { depth: 3 }).verdict).toBe('stop')
})

it('lists people newly placed and no longer placed, and leaves unsure paths out of the count', () => {
const after = base.map((p, i) => (i === 0 ? null : i === 1 ? 'M.?' : p))
const r = compareRuns(run([...base, null]), run([...after, 'P.1']))
expect(r.noLongerPlaced).toEqual(['m0', 'm1'])
expect(r.newlyPlaced).toEqual(['m200'])
expect(r.compared).toBe(198)
})

it('declines a verdict on too few people, and refuses runs of different kits', () => {
expect(compareRuns(run(base.slice(0, 20)), run(base.slice(0, 20))).verdict).toBe('too-few')
expect(() => compareRuns(run(base, 'A'), run(base, 'B'))).toThrow(/different kits/)
})

it('a planted kit rerun with a few matches gone stays stable', async () => {
const pk = await plantedKit(1, 0, 0.7)
const before = clusterKit(pk.imported.observations, pk.imported.links)
const rng = new SeededRandom(5)
const gone = new Set(rng.shuffle(before.matches.map((m) => m.match)).slice(0, 3))
const after = clusterKit(pk.imported.observations.filter((o) => !gone.has(o.match)), pk.imported.links.filter((l) => !gone.has(l.a) && !gone.has(l.b)))
// This family is small: 45 people are placed two levels deep.
const r = compareRuns(before, after, { minCompared: 20 })
expect(r.verdict).toBe('stable')
expect(r.noLongerPlaced.every((m) => gone.has(m))).toBe(true)
})
})
Loading