Add upstream-shaped scoped introspection plugin

[Zetazzz]

Summary

This is an upstream-shaped, opt-in alternative to #1719. It copies the exact Crystal release sources pinned by CNC ([email protected], [email protected]) and keeps the upstream package boundaries while adding a scoped query and replacement introspection plugin.

Unset/false uses untouched upstream stock introspection. GRAPHILE_SCOPED_INTROSPECTION=true dynamically installs ScopedIntrospectionPreset; GRAPHILE_SCOPED_INTROSPECTION_JIT independently controls the introspection session JIT setting and defaults to false.

Architecture and configuration

• pg-introspection: parameterized recursive schema/dependency-closure query, all/dependency-closure catalog type policy, and extension capability retention.
• graphile-build-pg: complete CNC-owned PgScopedIntrospectionPlugin, upstream lifecycle/watch/entity behavior, namespace and missing-type fail-closed guards, and pinned upstream contract sentinel.
• Native Graphile configuration: scoped options live in preset.gather, keyed by PostgreSQL service name. PG service objects are no longer mutated or extended with CNC-only fields.
• CNC wiring: the public boolean/env model remains unchanged; the server converts enabled CNC options into gather.pgScopedIntrospection and keeps JIT in pgSettingsForIntrospection.
• Stock startup does not load the scoped package, and ConstructivePreset remains unchanged.

Equivalent direct configuration:

const preset = {
  extends: [ScopedIntrospectionPreset],
  pgServices: [
    makePgService({
      name: "main",
      schemas: ["app_public"],
      pgSettingsForIntrospection: { jit: "off" },
    }),
  ],
  gather: {
    pgScopedIntrospection: {
      main: {
        catalogTypes: "dependency-closure",
        allowedDependencySchemas: ["app_private"],
        capabilityExtensions: ["pg_trgm"],
      },
    },
  },
};

Presence of a service entry enables scoped introspection for that service. Services without an entry remain stock, mixed stock/scoped services are supported, and an unknown configured service fails closed.

Benchmark arms differ only by upstream PgIntrospectionPlugin/stock query versus ScopedIntrospectionPreset/scoped query. They do not load ConstructivePreset, retirement/disposers, governor/cache limits, routing, or admission plugins.

Correctness coverage

The real PostgreSQL dependency-closure fixture covers cross-schema enum/domain/composite/range/multirange types, function signatures, ordinary and pg_trgm indexes, extension metadata, a cross-schema foreign key, and inheritance direction. It proves the required dependency parent is retained while a reverse parent-to-child relationship does not pull an unrelated child into the closure. No production query-algorithm change was needed for this configuration migration.

Additional contracts cover mixed stock/scoped services, stock replacement equivalence, persistent gather cache behavior, unknown-service rejection, invalid dependency schemas, missing-type rejection, schema equality, and runtime execution.

Catalog benchmark

The scoped-only benchmark reuses the generic fresh-process runner/protocol/report. It uses a fixed 65-table root surface plus dependency and unrelated noise schemas, scalable catalog noise, identical explicit JIT settings, public gather-hook entity counts, and a separate query-only worker. Every arm used 10 fresh-process samples and all 240 worker PIDs were unique.

Environment: macOS arm64, Node v22.22.0, PostgreSQL 18.4 in a dedicated local container. PostgreSQL catalog cache is labeled shared-server-not-reset; this is not a pristine-cache cold-start claim.

JIT off (primary)

Negative percentages mean scoped reduced the metric.

Size	pg_class	Build ms stock → scoped p50/p95	Paired Δ	Query ms stock → scoped p50/p95	Paired Δ	JSON bytes stock → scoped
small	825	322/333 → 329/342	+1.9%	56/60 → 60/66	+6.0%	1,372,051 → 1,060,547
medium	5,631	626/669 → 341/347	-45.8%	270/277 → 64/71	-76.5%	13,684,742 → 1,060,616
large	65,037	4,682/5,066 → 398/430	-91.5%	3,116/3,214 → 111/147	-96.5%	163,949,368 → 1,060,547

At large scale, parsed classes fell from 64,220 to 270 and median heap fell from 839.6 MiB to 46.7 MiB. The small fixture shows why the feature remains opt-in.

JIT diagnostic

With JIT enabled, compilation dominates the recursive query at small/medium scale. The primary comparison therefore uses identical JIT-off sessions, matching CNC's scoped default. The benchmark reports the requested and observed JIT value for every run.

Stock/scoped schema hashes matched at every size and every runtime verification query passed:

• small: d31e8b1c2d9cab3d1e67dcbc734e5addd237637028089221cf5f014b9ede4aa5
• medium: 8c4e2e03008c98295978712461b799d23957ae89071326a16421ac4a7d0b1996
• large: 41c6ed8feb087766680ab1dc4d833e5082dbf69322b5fb2a118515e7c7ff9db0

Reproduction

pnpm --filter @constructive-io/perf-harness build
node packages/perf-harness/dist/scoped-catalog.js prepare --database-url <url> --fixture <unique-cperf-name> --size <small|medium|large>
node packages/perf-harness/dist/scoped-catalog.js run --database-url <url> --fixture <same-name> --size <same-size> --repetitions 10 --seed 20260819 --output-directory packages/perf-harness/artifacts/<name>

Fixture preparation never replaces or drops schemas. The large synthetic result demonstrates scale behavior; it is not a claim about typical production gains.

Validation

Passing:

• graphile-scoped-introspection: 30 tests / 8 suites, including 6 real PostgreSQL dependency-closure cases
• scoped graphile-settings wiring/JIT contracts: 3 tests
• GraphQL server stock/scoped wiring: 5 tests
• performance harness: 18 tests / 9 suites
• CJS and ESM builds for scoped introspection, GraphQL types, graphile-settings, GraphQL server, and perf-harness
• affected ESLint with no broad autofix
• pnpm install --frozen-lockfile
• git diff --check and PR-wide format-only diff audit

No database URL or password appears in benchmark JSON output.

[tenki-reviewer]

Review complete. 🟡 1 medium

💬 Inline comments (1)

• 🟡 capabilityExtensions schema rejected by assertScopedNamespaces — PgIntrospectionPlugin.ts:299

---

This PR vendors a scoped PostgreSQL introspection engine (dependency-closure CTE, ACL parsing, schema validation, watch/cache lifecycle) into a new graphile-scoped-introspection package and wires it through graphile-settings, the graphql/env option merge, the server middleware, and a new perf-harness benchmark suite with its own CI batch. The change is largely additive and well-tested, but one medium finding surfaced in the core plugin.

Files	Change
graphile/graphile-scoped-introspection/*	New vendored scoped introspection engine: dependency-closure query, ACL/namespace validation, cache and watch lifecycle
graphile/graphile-settings/*	New introspection-settings wiring exposing scoped introspection options
graphql/env, graphql/types, graphql/server	New env parsing/merge for scoped introspection options and middleware wiring
packages/perf-harness/*	New benchmark suite: catalog fixtures, analysis, query workers, suite runner
.github/workflows/run-tests.yaml	CI batch routing for the new package

The primary defect: capabilityExtensions adds the extension's schema to the scoped query's namespaces but allowedSchemas never includes it, so assertScopedNamespaces hard-fails the gather unless the schema is also listed in introspectionDependencySchemas. Several lower-confidence candidate findings (single-flight coalescing gap from the async buildPreset, fixture-name poisoning on mid-prepare failure, env boolean coercion, whitespace validation) fell below the confidence threshold.

_{Reviewed commit: 4d35ddb}

[tenki-reviewer]

Adds a CNC-owned scoped PostgreSQL introspection replacement (graphile-scoped-introspection), wiring it into graphile-settings, graphql/env config parsing, the GraphQL server middleware, and a new perf-harness benchmark package.

Key findings

• 🟡 capabilityExtensions schema rejected by assertScopedNamespaces — PgIntrospectionPlugin.ts:299

[tenki-reviewer]

🟡 bug · medium

capabilityExtensions schema rejected by assertScopedNamespaces

capabilityExtensions pulls the extension's schema into scoped_namespaces (scopedIntrospection.ts:328-330), but allowedSchemas (PgIntrospectionPlugin.ts:299-301) never includes it, so assertScopedNamespaces throws crossed into unapproved dependency schema and the gather hard-fails. Setting introspectionCapabilityExtensions without also listing the extension's schema in introspectionDependencySchemas breaks schema build entirely, making the option ineffective for its stated purpose.

📋 Prompt for AI Agents

In graphile/graphile-scoped-introspection/src/upstream/graphile-build-pg/PgIntrospectionPlugin.ts around lines 294-303, extend the allowedSchemas array to also include the namespaces of the configured capability extensions (resolve each capability extension's extnamespace from the introspected extensions/namespaces), so that assertScopedNamespaces in scopedValidation.ts does not reject the extension's schema. Currently capabilityExtensions (line 296) adds the extension's schema to scoped_namespaces in the SQL CTE, but allowedSchemas (lines 299-301) omits it, causing a hard 'crossed into unapproved dependency schema' failure whenever a capability extension lives in a schema not already listed in introspectionDependencySchemas.