diff --git a/docs/feature-flags.md b/docs/feature-flags.md index 457e9b12b..47f7a8e04 100644 --- a/docs/feature-flags.md +++ b/docs/feature-flags.md @@ -56,7 +56,8 @@ it can never ship a flag that silently resolves to `undefined`. > **Every `example-*` key in this document is illustrative, not a real flag.** > Grep for one and you will find only this file. The registry holds exactly the -> flags that gate live behaviour — today four rollout gates, all plain booleans: +> flags that gate live behaviour — today four rollout gates, all plain booleans +> (`insights-visual` also declares a 50/50 experiment that is not running): > `retention-policies`, `lore-explorer-instruments`, > `lore-explorer-duplicate-clusters` and `insights-visual`. It has also held entries since removed: the > rollout gate for the `/insights` route (deleted once that page became @@ -154,7 +155,7 @@ stamps the **active span** with: | `feature_flag.provider.name` | `lorekit-flags` | | | `feature_flag.context.id` | `user_abc123` | The `targetingKey`, when the evaluation context carries one | | `feature_flag.result.variant` | `treatment` | **This is the A/B dimension** — filter or group by it | -| `feature_flag.result.reason` | `SPLIT` | `SPLIT` = experiment bucketing ran; `STATIC` = no active experiment | +| `feature_flag.result.reason` | `split` | `split` = experiment bucketing ran; `static` = no active experiment; `override` = a session override won. Lowercased from OpenFeature's `SPLIT`/`STATIC` by `flagReasonValue` | ...plus a `feature_flag.evaluation` span event carrying the same data (so a trace that evaluates several flags still shows each one on the timeline), and @@ -209,6 +210,15 @@ for the rest of the session: | Attribute | Example | Notes | |-----------|---------|-------| | `feature_flag.` | `feature_flag.example-onboarding-flow = "treatment"` | One dynamically-named attribute PER FLAG, holding its variant | +| `lorekit.feature_flag.reason.` | `lorekit.feature_flag.reason.example-onboarding-flow = "split"` | How that variant was decided — the same values as the span's `feature_flag.result.reason`. Filter on `split` to keep developer overrides out of an experiment | + +Each sync REPLACES the previous values (`removeSignalAttribute` then +`addSignalAttribute`): the SDK's add appends, so a re-sync after an override +flip used to leave both variants on every later signal. + +In PromQL a hyphen in the flag key becomes an underscore: +`feature_flag.insights-visual` is the label `feature_flag_insights_visual`, and +its reason is `lorekit_feature_flag_reason_insights_visual`. This is a genuinely different shape from the server-side span attributes above, on purpose: the OTel feature-flag semantic conventions @@ -222,11 +232,21 @@ representation that doesn't collide, at the cost of not being spec-standard. Only the variant is attached, never an `object`-typed flag's whole payload — same "prefer `variant` over `value`" reasoning as the span hook. -**This is what answers "did treatment convert better than control":** search -or group Web Events by `feature_flag.example-onboarding-flow`, and compare -whatever conversion event you're tracking (a `sign_up_completed` custom event, -a specific page reached) between the `"control"` and `"treatment"` -populations. +**This is what answers "did treatment convert better than control":** group +the conversion event you're tracking by `feature_flag.`, filtered to +`lorekit.feature_flag.reason. = "split"`, and divide by each arm's +exposed users from `feature_flag.exposed` — see +[Running an experiment](#running-an-experiment). + +**Known gap — the initial page load.** The sync runs after React mounts, so +signals the SDK records while a page first loads carry no flag tags: 192 of 219 +dashboard page views (88%), 115 of 151 navigation timings and all 31 browser +errors over 2026-10-14..21. `@dash0/sdk-web` only attaches attributes to those +when they are passed to `init()` as `additionalSignalAttributes`, and `init()` +runs before the flags are known. Until that changes, do not compare page views, +load-time web vitals or browser errors per variant; count the experiment's +population with `feature_flag.exposed`, which is emitted after mount and always +carries its variant. **Before this was wired (nothing appears):** if you evaluated flags server- side (Server Components) before `FeatureFlagsProvider` existed on this @@ -235,6 +255,122 @@ provider mounts, no `feature_flag.*` attribute reaches RUM at all — only the server-side span hook fires. A `feature_flag.*` search on Web Events finding nothing is the correct, expected symptom of that gap, not a broken query. +## Running an experiment + +Per-user splits need no storage: a signed-in user is bucketed by their Supabase +user id (see [Targeting key resolution](#targeting-key-resolution)), so they see +the same arm on every device for as long as the weights stand still. +`insights-visual` is declared this way and is the worked example. + +### 1. Declare the arms, dark + +```ts +experiment: { + enabled: false, + variants: [ + { key: 'on', weight: 50, description: 'The new surface' }, + { key: 'off', weight: 50, description: 'Control' }, + ], +}, +``` + +`enabled: false` resolves every user to `defaultVariant` with reason `static`, +so merging changes nothing. **List the treatment first**: the first arm owns +buckets `[0, weight)`, so raising its weight only ever adds users to it +(`bucketing.spec.ts` § "ramping an experiment"). Lowering it, or reordering the +arms, moves users between arms mid-experiment. + +### 2. Count exposure in the resolver + +Call `useFeatureFlagExposure('')` in the copy-and-suffix resolver, +never in one arm — the control arm's exposures are half the comparison. It emits +`feature_flag.exposed` once per mount (and again if the variant changes), one +task after the commit so it carries the real user id rather than the anonymous +one: + +| Attribute | Example | +|---|---| +| `feature_flag.key` | `insights-visual` | +| `feature_flag.result.variant` | `on` | +| `feature_flag.result.reason` | `split` | + +### 3. Launch and ramp + +Set `enabled: true` and deploy. To ramp, raise the first arm's weight (and lower +the second to keep the sum at 100). Visitors who are not signed in are bucketed +by the `lorekit_flag_anon_id` cookie and can change arm when they sign in, so +run experiments on authenticated surfaces. + +### 4. Measure + +Exposed users per arm — the population, and the sample-ratio check: + +```promql +sum by (feature_flag_result_variant) ( + clamp_max( + sum by (feature_flag_result_variant, user_id) ( + increase({otel_metric_name="dash0.web.events", service_name="web", + event_name="feature_flag.exposed", feature_flag_key="insights-visual", + feature_flag_result_reason="split", user_id!="", user_id!~"anon:.*"}[$__range]) + ), + 1 + ) +) +``` + +The clamped sum is the distinct-user shape that matches a SQL `uniqExact` +on `dash0.web.events`; `count()` and `dash0.web.users` do not. + +**Sample-ratio mismatch is the regression signal.** With `n` exposed users and an +arm weighted `p`, that arm's share should stay within `p ± 3·√(p(1−p)/n)` — +±15 points at 100 users, ±7.5 at 400. Outside it, assignment or exposure is +broken (a resolver that only one arm reaches, a cached page): stop reading +outcomes until it is explained. + +Outcomes per exposed user, per arm, overrides excluded — here every Insights +interaction. The exposure event names its arm `feature_flag_result_variant` and +the outcome events `feature_flag_insights_visual`, so the denominator is +relabelled before dividing: + +```promql +sum by (feature_flag_insights_visual) ( + increase({otel_metric_name="dash0.web.events", service_name="web", + event_name=~"^insights[.].*", lorekit_feature_flag_reason_insights_visual="split"}[$__range]) +) +/ on (feature_flag_insights_visual) +sum by (feature_flag_insights_visual) ( + label_replace( + clamp_max( + sum by (feature_flag_result_variant, user_id) ( + increase({otel_metric_name="dash0.web.events", service_name="web", + event_name="feature_flag.exposed", feature_flag_key="insights-visual", + feature_flag_result_reason="split", user_id!="", user_id!~"anon:.*"}[$__range]) + ), + 1 + ), + "feature_flag_insights_visual", "$1", "feature_flag_result_variant", "(.*)" + ) +) +``` + +Server-side outcomes (errors, latency) group spans by +`feature_flag.result.variant` instead. + +### 5. Finish + +Ship the winner as [Removing a flag](#removing-a-flag) describes: promote its +arm into the resolver, delete the other, drop the registry entry — and the +`useFeatureFlagExposure` call with it. + +### What this does not do + +- **No saved per-user assignment.** The arm is a pure function of flag key, + user id and weights. There is no way to pin one user to an arm across devices + (a session override pins one browser, and only for allowlisted developers in + production), and a non-monotonic weight change reassigns users. +- **Weights change by deploy.** `registry.ts` stays the single source of truth; + there is no runtime percentage control. + ## Cross-language flags (adding a language) Nothing above is TypeScript-specific in its _data_. `generated/flags.manifest.json` diff --git a/docs/otel.md b/docs/otel.md index be24f8597..dea488bf2 100644 --- a/docs/otel.md +++ b/docs/otel.md @@ -1075,6 +1075,18 @@ Both the command and the surface are reported as **bounded ids**, never the command string: `CopyCommand` takes arbitrary text, which would become unbounded the moment a call site interpolates into it. +### Feature flags (`feature_flag.*`) + +| Event | Emitted when | Attributes | +|-------|--------------|------------| +| `feature_flag.exposed` | A user reached a flag-gated surface, in either arm (once per mount, from the flag's resolver) | `feature_flag.key`, `feature_flag.result.variant`, `feature_flag.result.reason` (`static`/`split`/`override`) | + +The semantic-convention names rather than `lorekit.*`, because one event is one +evaluation's outcome — what those names describe — and the server-side span hook +uses the same three. Every RUM signal additionally carries +`feature_flag.` (variant) and `lorekit.feature_flag.reason.`; +see `docs/feature-flags.md` § Running an experiment. + --- ## Browser auth events (`auth.*`) diff --git a/packages/feature-flags/generated/flags.manifest.json b/packages/feature-flags/generated/flags.manifest.json index 02bf2cc31..5d3266310 100644 --- a/packages/feature-flags/generated/flags.manifest.json +++ b/packages/feature-flags/generated/flags.manifest.json @@ -57,20 +57,35 @@ }, { "key": "insights-visual", - "description": "Insights — the visual first version of `/insights`: ONE page-wide window, the Learn → Recall → Promote → Prune loop strip, the Pulse timeline (lessons written and records read on a shared time axis, chosen vs bulk), the recall flow (client → tool → scope type), and runs drawn as a weekday × hour rhythm plus swimlanes — with the existing 'earning its place' grid, runs list and operational health kept below. Gates the whole page body through the `InsightsView` resolver; with it off `/insights` renders exactly as before. Reads only routes that already ship (`/memories/activity`, `/read-activity`, `/usage`, `/usage/runs`, `/utility`), so there is no backend prerequisite and flipping it is a pure UI decision.", + "description": "Insights — the visual first version of `/insights`: ONE page-wide window, the Learn → Recall → Promote → Prune loop strip, the Pulse timeline (lessons written and records read on a shared time axis, chosen vs bulk), the recall flow (client → tool → scope type), and runs drawn as a weekday × hour rhythm plus swimlanes — with the existing 'earning its place' grid, runs list and operational health kept below. Gates the whole page body through the `InsightsView` resolver; with it off `/insights` renders exactly as before. Reads only routes that already ship (`/memories/activity`, `/read-activity`, `/usage`, `/usage/runs`, `/utility`), so there is no backend prerequisite and flipping it is a pure UI decision. Declared as a 50/50 experiment that is NOT running (`enabled: false` resolves every user to `off`): launching it is flipping that one boolean, and `on` is listed first so ramping its weight up never moves a user from `on` back to `off`. Exposure is measured by `feature_flag.exposed`, emitted by the `InsightsView` resolver — see docs/feature-flags.md § Running an experiment.", "type": "boolean", "variants": { "off": false, "on": true }, "defaultVariant": "off", - "experiment": null, + "experiment": { + "enabled": false, + "variants": [ + { + "key": "on", + "description": "The visual page", + "weight": 50 + }, + { + "key": "off", + "description": "The original page (control)", + "weight": 50 + } + ] + }, "owner": "@lorekit/web", "tags": [ "dashboard", "web", "insights", - "rollout" + "rollout", + "experiment" ] } ] diff --git a/packages/feature-flags/src/bucketing.spec.ts b/packages/feature-flags/src/bucketing.spec.ts index 471382d86..cd2711be0 100644 --- a/packages/feature-flags/src/bucketing.spec.ts +++ b/packages/feature-flags/src/bucketing.spec.ts @@ -83,3 +83,29 @@ describe('assignExperimentVariant', () => { expect(sawDifference).toBe(true); }); }); + +describe('ramping an experiment', () => { + it('never moves a user from the first arm to the second when the first arm only grows', () => { + // Cumulative bucketing: the first arm owns buckets [0, w). Raising w keeps + // every bucket it already owned, so a ramp from 10% to 50% only ADDS users + // to the treatment — the reason the registry lists `on` first. + const ramp = [10, 25, 50, 100].map( + (onWeight): Experiment => ({ + enabled: true, + variants: [ + { key: 'on', weight: onWeight }, + { key: 'off', weight: 100 - onWeight }, + ], + }), + ); + for (let i = 0; i < 2_000; i++) { + const user = `user-${i}`; + let wasOn = false; + for (const stage of ramp) { + const isOn = assignExperimentVariant('insights-visual', stage, user) === 'on'; + if (wasOn) expect(isOn).toBe(true); + wasOn = isOn; + } + } + }); +}); diff --git a/packages/feature-flags/src/client.spec.ts b/packages/feature-flags/src/client.spec.ts index 1c01a1704..372920a93 100644 --- a/packages/feature-flags/src/client.spec.ts +++ b/packages/feature-flags/src/client.spec.ts @@ -20,6 +20,8 @@ */ import { beforeEach, describe, expect, it } from 'vitest'; import { evaluateFlag, evaluateFlagDetails, resetFeatureFlagClientForTests } from './client.ts'; +import { FLAG_REGISTRY } from './registry.ts'; +import type { FlagKey } from './generated/flags.generated.ts'; describe('evaluateFlag', () => { beforeEach(() => { @@ -62,4 +64,17 @@ describe('evaluateFlagDetails', () => { }); expect(details).toMatchObject({ value: false, variant: 'off', reason: 'OVERRIDE' }); }); + + it('keeps every declared-but-disabled experiment on its default for every user', async () => { + // A dark experiment must change nothing: no user lands on another arm and + // none is reported as SPLIT. Derived from the registry rather than naming a + // flag, so launching one is still a single `enabled: true` edit. + const dark = FLAG_REGISTRY.filter((flag) => flag.experiment && !flag.experiment.enabled); + for (const flag of dark) { + for (const targetingKey of ['user-a', 'user-b', 'user-c', 'user-d']) { + const details = await evaluateFlagDetails(flag.key as FlagKey, { targetingKey }); + expect(details).toMatchObject({ variant: flag.defaultVariant, reason: 'STATIC' }); + } + } + }); }); diff --git a/packages/feature-flags/src/index.ts b/packages/feature-flags/src/index.ts index 113435369..aefa9a566 100644 --- a/packages/feature-flags/src/index.ts +++ b/packages/feature-flags/src/index.ts @@ -50,6 +50,7 @@ export { ATTR_FEATURE_FLAG_RESULT_REASON, EVENT_FEATURE_FLAG_EVALUATION, METRIC_FEATURE_FLAG_EVALUATIONS, + flagReasonValue, } from './otel-attributes.ts'; export type { FlagKey, FlagValue, FlagValueMap } from './generated/flags.generated.ts'; export { FLAG_KEYS } from './generated/flags.generated.ts'; diff --git a/packages/feature-flags/src/otel-attributes.spec.ts b/packages/feature-flags/src/otel-attributes.spec.ts new file mode 100644 index 000000000..e681629fb --- /dev/null +++ b/packages/feature-flags/src/otel-attributes.spec.ts @@ -0,0 +1,16 @@ +import { describe, expect, it } from 'vitest'; +import { flagReasonValue } from './otel-attributes.ts'; + +describe('flagReasonValue', () => { + it("lowercases OpenFeature's reasons to the semantic conventions' values", () => { + expect(flagReasonValue('STATIC')).toBe('static'); + expect(flagReasonValue('SPLIT')).toBe('split'); + expect(flagReasonValue('OVERRIDE')).toBe('override'); + }); + + it('reports a missing reason as unknown, never an empty string', () => { + // An empty attribute value is indistinguishable from "not set" in a query. + expect(flagReasonValue(undefined)).toBe('unknown'); + expect(flagReasonValue('')).toBe('unknown'); + }); +}); diff --git a/packages/feature-flags/src/otel-attributes.ts b/packages/feature-flags/src/otel-attributes.ts index ceb37a1c1..d1f45811f 100644 --- a/packages/feature-flags/src/otel-attributes.ts +++ b/packages/feature-flags/src/otel-attributes.ts @@ -37,3 +37,18 @@ export const EVENT_FEATURE_FLAG_EVALUATION = 'feature_flag.evaluation'; /** `lorekit.feature_flag.evaluations` — counter dimensioned by key + variant. */ export const METRIC_FEATURE_FLAG_EVALUATIONS = 'lorekit.feature_flag.evaluations'; + +/** + * A resolution reason as a `feature_flag.result.reason` VALUE: OpenFeature's + * UPPERCASE reasons lowercased. "STATIC" and "SPLIT" become the semantic + * conventions' well-known `static` and `split`; this package's own "OVERRIDE" + * becomes `override`, a custom value the conventions permit but do not define. + * + * One function, because three surfaces carry the reason — server spans, the + * browser's per-flag RUM attribute, and the exposure event — and an experiment + * query filters all three with ONE vocabulary. An absent reason is `unknown`, + * never an empty string, so a query can still tell "not reported" from a value. + */ +export function flagReasonValue(reason: string | undefined): string { + return reason ? reason.toLowerCase() : 'unknown'; +} diff --git a/packages/feature-flags/src/otel-hook.ts b/packages/feature-flags/src/otel-hook.ts index 00953b289..81017c353 100644 --- a/packages/feature-flags/src/otel-hook.ts +++ b/packages/feature-flags/src/otel-hook.ts @@ -28,6 +28,7 @@ import { ATTR_FEATURE_FLAG_RESULT_VARIANT, EVENT_FEATURE_FLAG_EVALUATION, METRIC_FEATURE_FLAG_EVALUATIONS, + flagReasonValue, } from './otel-attributes.ts'; import { getFeatureFlagMeter } from './telemetry.ts'; @@ -66,13 +67,10 @@ export const featureFlagOtelHook: Hook = { const span = trace.getActiveSpan(); const attrs = baseAttributes(hookContext); if (details.variant) attrs[ATTR_FEATURE_FLAG_RESULT_VARIANT] = details.variant; - // Lowercased: OpenFeature's own `StandardResolutionReasons` values are - // UPPERCASE ("STATIC", "SPLIT") by its own convention, but the OTel - // feature-flag semantic conventions' well-known `feature_flag.result.reason` - // values are lowercase ("static", "split") — see the spec link in - // `otel-attributes.ts`. Transform at this boundary only; `provider.ts` keeps - // the idiomatic OpenFeature casing for its own `ResolutionDetails.reason`. - if (details.reason) attrs[ATTR_FEATURE_FLAG_RESULT_REASON] = details.reason.toLowerCase(); + // Lowercased by `flagReasonValue`: OpenFeature's reasons are UPPERCASE, the + // semantic conventions' values lowercase. Transform at this boundary only; + // `provider.ts` keeps the idiomatic OpenFeature casing. + if (details.reason) attrs[ATTR_FEATURE_FLAG_RESULT_REASON] = flagReasonValue(details.reason); if (span) stampSpan(span, attrs); recordEvaluationMetric(hookContext.flagKey, details.variant); }, diff --git a/packages/feature-flags/src/registry.ts b/packages/feature-flags/src/registry.ts index b215f83ea..51f600e0c 100644 --- a/packages/feature-flags/src/registry.ts +++ b/packages/feature-flags/src/registry.ts @@ -81,12 +81,19 @@ const REGISTRY_INPUT = [ { key: 'insights-visual', description: - "Insights — the visual first version of `/insights`: ONE page-wide window, the Learn → Recall → Promote → Prune loop strip, the Pulse timeline (lessons written and records read on a shared time axis, chosen vs bulk), the recall flow (client → tool → scope type), and runs drawn as a weekday × hour rhythm plus swimlanes — with the existing 'earning its place' grid, runs list and operational health kept below. Gates the whole page body through the `InsightsView` resolver; with it off `/insights` renders exactly as before. Reads only routes that already ship (`/memories/activity`, `/read-activity`, `/usage`, `/usage/runs`, `/utility`), so there is no backend prerequisite and flipping it is a pure UI decision.", + "Insights — the visual first version of `/insights`: ONE page-wide window, the Learn → Recall → Promote → Prune loop strip, the Pulse timeline (lessons written and records read on a shared time axis, chosen vs bulk), the recall flow (client → tool → scope type), and runs drawn as a weekday × hour rhythm plus swimlanes — with the existing 'earning its place' grid, runs list and operational health kept below. Gates the whole page body through the `InsightsView` resolver; with it off `/insights` renders exactly as before. Reads only routes that already ship (`/memories/activity`, `/read-activity`, `/usage`, `/usage/runs`, `/utility`), so there is no backend prerequisite and flipping it is a pure UI decision. Declared as a 50/50 experiment that is NOT running (`enabled: false` resolves every user to `off`): launching it is flipping that one boolean, and `on` is listed first so ramping its weight up never moves a user from `on` back to `off`. Exposure is measured by `feature_flag.exposed`, emitted by the `InsightsView` resolver — see docs/feature-flags.md § Running an experiment.", type: 'boolean', variants: { off: false, on: true }, defaultVariant: 'off', + experiment: { + enabled: false, + variants: [ + { key: 'on', weight: 50, description: 'The visual page' }, + { key: 'off', weight: 50, description: 'The original page (control)' }, + ], + }, owner: '@lorekit/web', - tags: ['dashboard', 'web', 'insights', 'rollout'], + tags: ['dashboard', 'web', 'insights', 'rollout', 'experiment'], }, ] satisfies unknown[]; diff --git a/packages/web/src/app/(dashboard)/layout.tsx b/packages/web/src/app/(dashboard)/layout.tsx index dfb83117b..6a35356f1 100644 --- a/packages/web/src/app/(dashboard)/layout.tsx +++ b/packages/web/src/app/(dashboard)/layout.tsx @@ -71,16 +71,17 @@ export default async function DashboardLayout({ children }: { children: React.Re // Evaluated ONCE here, server-side, for the whole dashboard tree. // `FeatureFlagsProvider` hands `values` to every Client Component via - // `useFeatureFlag`, and forwards `variants` into RUM (`dash0-rum.ts`) so - // Web Events can be filtered/grouped by `feature_flag.` — there is no + // `useFeatureFlag`, and forwards `variants` and `reasons` into RUM + // (`dash0-rum.ts`) so Web Events can be filtered/grouped by + // `feature_flag.` and how it was assigned — there is no // separate client-side evaluation to drift from this one. `user.id` is // passed through so this does not repeat the `auth.getUser()` call // `resolveDashboardBootstrap` already made above. See // `lib/feature-flags/server.ts`. - const { values: flags, variants: flagVariants } = await getAllServerFlagState(user.id); + const { values: flags, variants: flagVariants, reasons: flagReasons } = await getAllServerFlagState(user.id); return ( - + {/* CommandPaletteProvider wraps the entire dashboard so the palette is diff --git a/packages/web/src/components/insights/InsightsView.tsx b/packages/web/src/components/insights/InsightsView.tsx index 20f4993ec..e396e0f99 100644 --- a/packages/web/src/components/insights/InsightsView.tsx +++ b/packages/web/src/components/insights/InsightsView.tsx @@ -19,12 +19,14 @@ * registry's `defaultVariant: 'off'` so the two cannot disagree. */ -import { useFeatureFlagVariant } from '@/components/providers/FeatureFlagsProvider'; +import { useFeatureFlagExposure, useFeatureFlagVariant } from '@/components/providers/FeatureFlagsProvider'; import { InsightsViewOff } from './InsightsView.off'; import { InsightsViewOn } from './InsightsView.on'; export function InsightsView() { const variant = useFeatureFlagVariant('insights-visual'); + // In the resolver, so both arms are counted — see `useFeatureFlagExposure`. + useFeatureFlagExposure('insights-visual'); switch (variant) { case 'on': return ; diff --git a/packages/web/src/components/providers/FeatureFlagsProvider.tsx b/packages/web/src/components/providers/FeatureFlagsProvider.tsx index 699cc25b6..b180d192e 100644 --- a/packages/web/src/components/providers/FeatureFlagsProvider.tsx +++ b/packages/web/src/components/providers/FeatureFlagsProvider.tsx @@ -28,19 +28,29 @@ * (`server.ts`) — is forwarded to `syncFeatureFlagRumAttributes` * (`lib/dash0-rum.ts`) on mount and whenever it changes, so every subsequent * RUM signal (page view, click, custom event) in this session carries - * `feature_flag.` = variant. That is what makes "which flags were active + * `feature_flag.` = variant, and `reasons` beside it as + * `lorekit.feature_flag.reason.` = `static` | `split` | `override`. That is what makes "which flags were active * for this visitor" and "which experiment arm" answerable retrospectively * from the Web Events explorer — see `docs/feature-flags.md` § * "Feature flags in telemetry" for why this is a different mechanism than the * server-side OTel span hook (`otel-hook.ts`). + * + * ## Exposure: who actually reached a flagged surface + * + * Every RUM signal carries every flag, so "carries `feature_flag.x = on`" is not + * "saw the `on` arm". {@link useFeatureFlagExposure}, called from a flag's + * resolver, records `feature_flag.exposed` once per mount in BOTH arms — the + * population an experiment compares. */ import { createContext, useContext, useEffect, type ReactNode } from 'react'; import type { FlagKey, FlagValueMap } from '@lorekit/feature-flags'; import { syncFeatureFlagRumAttributes } from '@/lib/dash0-rum'; +import { track } from '@/lib/analytics/track'; interface FeatureFlagsContextValue { values: FlagValueMap; variants: Readonly>; + reasons: Readonly>; } const FeatureFlagsContext = createContext(null); @@ -50,24 +60,26 @@ export interface FeatureFlagsProviderProps { flags: FlagValueMap; /** Flag key -> variant, from the SAME evaluation `flags` came from. */ variants: Readonly>; + /** Flag key -> how that variant was decided (`static`, `split`, `override`), from the same evaluation. */ + reasons: Readonly>; children: ReactNode; } /** Mount once near the dashboard root, seeded with a server-evaluated flag map. */ -export function FeatureFlagsProvider({ flags, variants, children }: FeatureFlagsProviderProps) { +export function FeatureFlagsProvider({ flags, variants, reasons, children }: FeatureFlagsProviderProps) { // Keyed on a stable string, not the `variants` object identity — the // dashboard layout builds a fresh object every render, and re-tagging RUM // signals with identical content on every navigation would be pure churn. - const variantsKey = JSON.stringify(variants); + const stateKey = JSON.stringify([variants, reasons]); useEffect(() => { - syncFeatureFlagRumAttributes(variants); - // `variantsKey` (not `variants`) is the intentional dependency — content-based, - // since the dashboard layout builds a fresh `variants` object every render. + syncFeatureFlagRumAttributes(variants, reasons); + // `stateKey` (not the objects) is the intentional dependency — content-based, + // since the dashboard layout builds fresh objects every render. // eslint-disable-next-line react-hooks/exhaustive-deps - }, [variantsKey]); + }, [stateKey]); return ( - + {children} ); @@ -108,3 +120,36 @@ export function useFeatureFlag(key: K): FlagValueMap[K] { export function useFeatureFlagVariant(key: FlagKey): string { return useFeatureFlagsContext().variants[key]; } + +/** + * How one flag's variant was decided — `static` (the default), `split` (an + * experiment assigned it) or `override` (a developer forced it) — spelled as + * server spans spell `feature_flag.result.reason`. `unknown` when the + * evaluation reported none. + */ +export function useFeatureFlagReason(key: FlagKey): string { + return useFeatureFlagsContext().reasons[key] ?? 'unknown'; +} + +/** + * Record that this user reached the surface `key` gates: one + * `feature_flag.exposed` event per mount, and again if the variant changes. + * + * Call it from the flag's RESOLVER, not from one arm — the control arm's + * exposures are half of the comparison. + * + * Deferred one task on purpose. React runs a child's effects before its + * parents', so an immediate event would leave before this provider's own RUM + * sync — the provider wraps the page — and miss the per-flag attributes. One + * `setTimeout` lands after every effect of the commit, and so also after + * `Dash0Provider` has replaced the anonymous id with the user's in any order + * the tree might render them. + */ +export function useFeatureFlagExposure(key: FlagKey): void { + const variant = useFeatureFlagVariant(key); + const reason = useFeatureFlagReason(key); + useEffect(() => { + const timer = setTimeout(() => track({ name: 'feature_flag.exposed', flagKey: key, variant, reason }), 0); + return () => clearTimeout(timer); + }, [key, variant, reason]); +} diff --git a/packages/web/src/lib/analytics/track.spec.ts b/packages/web/src/lib/analytics/track.spec.ts index d7b052dac..0236305f7 100644 --- a/packages/web/src/lib/analytics/track.spec.ts +++ b/packages/web/src/lib/analytics/track.spec.ts @@ -522,3 +522,17 @@ describe('track — insights events', () => { }); }); }); + +describe('track — feature flags', () => { + it('maps feature_flag.exposed onto the semantic-convention flag attributes', () => { + track({ name: 'feature_flag.exposed', flagKey: 'insights-visual', variant: 'on', reason: 'split' }); + expect(lastCall()).toEqual({ + name: 'feature_flag.exposed', + attributes: { + 'feature_flag.key': 'insights-visual', + 'feature_flag.result.variant': 'on', + 'feature_flag.result.reason': 'split', + }, + }); + }); +}); diff --git a/packages/web/src/lib/analytics/track.ts b/packages/web/src/lib/analytics/track.ts index cef96c724..7110b754d 100644 --- a/packages/web/src/lib/analytics/track.ts +++ b/packages/web/src/lib/analytics/track.ts @@ -64,6 +64,15 @@ * - `insights.scope_opened` — a row handed off to the Explorer. * - `insights.recall_flow_view_changed` — the visual page's recall flow swapped between chart and table. * + * ### Feature flags (`feature_flag.*`) + * - `feature_flag.exposed` — a user reached a surface a flag gates, in either arm. + * An experiment's population: count it by variant AND reason, and drop + * `override` rows, or a developer's forced arm reads as random assignment. + * Its attributes are the semantic-convention names the server-side OTel hook + * stamps on spans (`feature_flag.key`, `.result.variant`, `.result.reason`), + * not `lorekit.*`: one event is one evaluation's outcome, exactly what those + * names describe, and one vocabulary then filters both signals. + * * ## PII / cardinality * Command ids are a fixed enum EXCEPT the dynamic "Open Lesson…" children, whose * ids embed `scope::key` (user content + unbounded cardinality). `normalizeCommandId` @@ -85,6 +94,7 @@ */ import { sendEvent } from '@dash0/sdk-web'; +import type { FlagKey } from '@lorekit/feature-flags'; import type { ButtonSize, ButtonVariant } from '@/lib/button-styles'; import type { FilterField, FilterOperator } from '@/lib/filters'; import type { LessonUtility } from '@/lib/lesson-utility'; @@ -369,7 +379,16 @@ export type AnalyticsEvent = succeeded: boolean; } | { name: 'insights.scope_opened'; scopeType: ScopeSelectionType; source: InsightsScopeSource } - | { name: 'insights.recall_flow_view_changed'; view: 'chart' | 'table' }; + | { name: 'insights.recall_flow_view_changed'; view: 'chart' | 'table' } + // ── Feature flags ───────────────────────────────────────────────────────── + | { + name: 'feature_flag.exposed'; + flagKey: FlagKey; + /** A registry variant key — bounded by the flag's own declaration. */ + variant: string; + /** `static` | `split` | `override` | … as `flagReasonValue` spells it. */ + reason: string; + }; /** * Dynamic lesson commands embed `scope::key` in their id (user content + @@ -595,6 +614,12 @@ function toAttributes(event: AnalyticsEvent): Record { }; case 'insights.recall_flow_view_changed': return { 'lorekit.insights.recall_flow.view': event.view }; + case 'feature_flag.exposed': + return { + 'feature_flag.key': event.flagKey, + 'feature_flag.result.variant': event.variant, + 'feature_flag.result.reason': event.reason, + }; } } diff --git a/packages/web/src/lib/dash0-rum.spec.ts b/packages/web/src/lib/dash0-rum.spec.ts index 793d08a6c..c933a1d95 100644 --- a/packages/web/src/lib/dash0-rum.spec.ts +++ b/packages/web/src/lib/dash0-rum.spec.ts @@ -49,6 +49,7 @@ const { installExtensionErrorFilter, syncFeatureFlagRumAttributes, FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX, + FEATURE_FLAG_REASON_RUM_ATTRIBUTE_PREFIX, } = await import('./dash0-rum'); const { __signalAttributes: signalAttributes } = (await import('@dash0/sdk-web')) as unknown as { @@ -292,19 +293,52 @@ describe('syncFeatureFlagRumAttributes', () => { }); it('attaches one feature_flag. attribute per entry, holding the variant', () => { - syncFeatureFlagRumAttributes({ - 'example-onboarding-flow': 'treatment', - 'example-usage-charts': 'off', - }); + syncFeatureFlagRumAttributes( + { + 'example-onboarding-flow': 'treatment', + 'example-usage-charts': 'off', + }, + {}, + ); expect(valuesOf(`${FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX}example-onboarding-flow`)).toEqual([ 'treatment', ]); expect(valuesOf(`${FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX}example-usage-charts`)).toEqual(['off']); }); + it("attaches each flag's assignment reason beside its variant", () => { + syncFeatureFlagRumAttributes( + { 'example-onboarding-flow': 'treatment' }, + { 'example-onboarding-flow': 'split' }, + ); + expect(valuesOf(`${FEATURE_FLAG_REASON_RUM_ATTRIBUTE_PREFIX}example-onboarding-flow`)).toEqual(['split']); + }); + + it('omits the reason attribute for a flag with no reported reason', () => { + syncFeatureFlagRumAttributes({ 'example-onboarding-flow': 'treatment' }, {}); + expect(valuesOf(`${FEATURE_FLAG_REASON_RUM_ATTRIBUTE_PREFIX}example-onboarding-flow`)).toEqual([]); + }); + + it('replaces the previous variant and reason on a re-sync instead of adding a second entry', () => { + // An override flip re-runs the sync; the SDK's append used to leave both + // `off` and `on` on every later signal — both arms at once. + syncFeatureFlagRumAttributes( + { 'example-onboarding-flow': 'off' }, + { 'example-onboarding-flow': 'static' }, + ); + syncFeatureFlagRumAttributes( + { 'example-onboarding-flow': 'on' }, + { 'example-onboarding-flow': 'override' }, + ); + expect(valuesOf(`${FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX}example-onboarding-flow`)).toEqual(['on']); + expect(valuesOf(`${FEATURE_FLAG_REASON_RUM_ATTRIBUTE_PREFIX}example-onboarding-flow`)).toEqual([ + 'override', + ]); + }); + it('does nothing for an empty flag map', () => { const before = signalAttributes.length; - syncFeatureFlagRumAttributes({}); + syncFeatureFlagRumAttributes({}, {}); expect(signalAttributes).toHaveLength(before); }); }); diff --git a/packages/web/src/lib/dash0-rum.ts b/packages/web/src/lib/dash0-rum.ts index 9377eb6ff..6ef3d6069 100644 --- a/packages/web/src/lib/dash0-rum.ts +++ b/packages/web/src/lib/dash0-rum.ts @@ -29,7 +29,7 @@ * Next.js evaluates outside the React tree, and also server-side during static * prerendering) can import it as safely as a client component can. */ -import { init, identify, addSignalAttribute } from '@dash0/sdk-web'; +import { init, identify, addSignalAttribute, removeSignalAttribute } from '@dash0/sdk-web'; import { resolveAnonymousId } from './anonymous-id'; import { shouldIgnoreErrorFromExtension, stackOfUnknown } from './extension-errors'; @@ -395,6 +395,16 @@ export function resetDash0Identity(): void { /** Attribute-name prefix for the per-flag RUM tags {@link syncFeatureFlagRumAttributes} sets. */ export const FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX = 'feature_flag.'; +/** + * Attribute-name prefix for each flag's assignment REASON (`static`, `split`, + * `override`), set beside its variant by {@link syncFeatureFlagRumAttributes}. + * + * Its own `lorekit.*` namespace rather than `feature_flag..reason`: the + * semantic conventions' naming rules say a name should not also be a namespace, + * and `feature_flag.` is already a leaf holding the variant. + */ +export const FEATURE_FLAG_REASON_RUM_ATTRIBUTE_PREFIX = 'lorekit.feature_flag.reason.'; + /** * Attach the current session's feature-flag state to every subsequent RUM * signal — page views, clicks, custom events, errors — via `addSignalAttribute`. @@ -438,14 +448,42 @@ export const FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX = 'feature_flag.'; * * Called from `FeatureFlagsProvider` (a `useEffect`, so it re-runs whenever * the server re-evaluates flags — e.g. after a session-override change) with - * the variant map `getAllServerFlagVariants()` (`lib/feature-flags/server.ts`) - * produced. No-op before RUM init, like every other function in this module — + * the variant and reason maps `getAllServerFlagState()` + * (`lib/feature-flags/server.ts`) produced. Each re-run REPLACES the previous + * values rather than adding to them. + * + * ## Known gap: the initial page load + * + * This runs after React mounts, so signals the SDK records while a page first + * loads — the page view, navigation timing, early web vitals and errors — carry + * no flag tags (88% of dashboard page views over 2026-10-14..21). Compare those + * per variant only once flags reach `init()`'s `additionalSignalAttributes`; for + * an experiment's population, use the `feature_flag.exposed` event instead. No-op before RUM init, like every other function in this module — * a flag evaluated during static prerendering (no browser, no SDK) has * nothing to attach to yet; the next real page load calls this again anyway. */ -export function syncFeatureFlagRumAttributes(flagVariants: Readonly>): void { +export function syncFeatureFlagRumAttributes( + flagVariants: Readonly>, + flagReasons: Readonly>, +): void { if (!initialized) return; for (const [flagKey, variant] of Object.entries(flagVariants)) { - addSignalAttribute(`${FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX}${flagKey}`, variant); + replaceSignalAttribute(`${FEATURE_FLAG_RUM_ATTRIBUTE_PREFIX}${flagKey}`, variant); + const reason = flagReasons[flagKey]; + if (reason !== undefined) { + replaceSignalAttribute(`${FEATURE_FLAG_REASON_RUM_ATTRIBUTE_PREFIX}${flagKey}`, reason); + } } } + +/** + * Set a signal attribute to exactly one value. + * + * `addSignalAttribute` APPENDS. A re-sync — after an override change, or React + * running the effect twice in development — would otherwise leave the old and + * the new variant on every later signal, which reads as both arms at once. + */ +function replaceSignalAttribute(name: string, value: string): void { + removeSignalAttribute(name); + addSignalAttribute(name, value); +} diff --git a/packages/web/src/lib/feature-flags/server.ts b/packages/web/src/lib/feature-flags/server.ts index 2b8876b9e..bf86dcb96 100644 --- a/packages/web/src/lib/feature-flags/server.ts +++ b/packages/web/src/lib/feature-flags/server.ts @@ -16,6 +16,7 @@ import { FLAG_KEYS, evaluateFlag, evaluateFlagDetails, + flagReasonValue, parseFlagOverrides, withFlagOverrides, type FlagKey, @@ -94,8 +95,8 @@ export async function getAllServerFlags(knownUserId?: string | null): Promise variant KEY string (`"treatment"`, not `true`) — is * what `FeatureFlagsProvider` forwards into @@ -105,13 +106,23 @@ export async function getAllServerFlags(knownUserId?: string | null): Promise how the variant was decided (`static`, `split`, + * `override`, spelled by `flagReasonValue` exactly as spans spell it) — is what + * lets an experiment's analysis keep a developer override out of its treatment + * arm: without it, an overridden `on` and a randomly assigned `on` are the same + * RUM value. + * * Uses `evaluateFlagDetails` (one call per flag, not two) so this never * double-evaluates — and never double-fires the OTel hook — relative to a * plain `getAllServerFlags()` call for the same request. */ export async function getAllServerFlagState( knownUserId?: string | null, -): Promise<{ values: FlagValueMap; variants: Record }> { +): Promise<{ + values: FlagValueMap; + variants: Record; + reasons: Record; +}> { const context = await resolveFeatureFlagContext(knownUserId); const entries = await Promise.all( FLAG_KEYS.map(async (key) => { @@ -122,15 +133,18 @@ export async function getAllServerFlagState( const values: Record = {}; const variants: Record = {}; + const reasons: Record = {}; for (const [key, details] of entries) { values[key] = details.value; // `variant` is optional per the OpenFeature spec, but `LoreKitFlagProvider` // always sets it (static/experiment/override all do) — the fallback only // guards a hypothetical future provider that doesn't. variants[key] = details.variant ?? String(details.value); + reasons[key] = flagReasonValue(details.reason); } return { values: values as unknown as FlagValueMap, variants: variants as Record, + reasons: reasons as Record, }; } diff --git a/packages/web/src/mocks/decorators.tsx b/packages/web/src/mocks/decorators.tsx index 6a10392b8..84d1a3f23 100644 --- a/packages/web/src/mocks/decorators.tsx +++ b/packages/web/src/mocks/decorators.tsx @@ -157,6 +157,11 @@ const DEFAULT_FLAG_VARIANTS: Readonly> = Object.fromEntri FLAG_REGISTRY.map((flag) => [flag.key, flag.defaultVariant]), ); +/** Every flag resolved by its default, as the real provider reports it outside an experiment. */ +const DEFAULT_FLAG_REASONS: Readonly> = Object.fromEntries( + FLAG_REGISTRY.map((flag) => [flag.key, 'static']), +); + /** * Seed `FeatureFlagsProvider` with the registry's defaults so components * reading `useFeatureFlag`/`useFeatureFlagVariant` (e.g. `LoreExplorer`'s @@ -165,7 +170,7 @@ const DEFAULT_FLAG_VARIANTS: Readonly> = Object.fromEntri * layout, which is the only place a real provider is mounted. */ export const withFeatureFlags: Decorator = (Story) => ( - + ); @@ -184,8 +189,11 @@ export const withFeatureFlags: Decorator = (Story) => ( export function withFlagVariants(overrides: Partial>): Decorator { const values: FlagValueMap = { ...DEFAULT_FLAG_VALUES }; const variants: Record = { ...DEFAULT_FLAG_VARIANTS }; + const reasons: Record = { ...DEFAULT_FLAG_REASONS }; for (const [key, variant] of Object.entries(overrides) as [FlagKey, string][]) { variants[key] = variant; + // A forced variant is what the real provider reports as an override. + reasons[key] = 'override'; // Every flag in the registry is `boolean`, so the value IS "is this the `on` // arm" — which is also why an unknown variant seeds `false`, matching the // resolver's `default → off`. The cast is needed because `FlagValueMap` is a @@ -193,7 +201,7 @@ export function withFlagVariants(overrides: Partial>): D (values as Record)[key] = variant === 'on'; } const Wrapped: Decorator = (Story) => ( - + );