Testing API: allow extensions to prioritize compatible TestRunProfiles

[bennycode]

The Testing API lets a TestItem be compatible with multiple tagged TestRunProfiles. When several compatible profiles are defaults, however, an extension cannot express which one VS Code should prefer for an automatic single-profile action such as the gutter Run button.

Motivation

This came up in vitest-dev/vscode#799 and the draft fix vitest-dev/vscode#805.

A test file can legitimately be included by more than one Vitest configuration:

vitest.config.base.ts # Root Test Profile (Base)
packages/foo/vitest.config.ts # Package Test Profile (extending Base)
packages/foo/test/example.test.ts # Test

The API-level shape is equivalent to:

const rootTag = new vscode.TestTag('root-config');
const packageTag = new vscode.TestTag('package-config');

const item = controller.createTestItem('example', 'example', testUri);
item.tags = [rootTag, packageTag];
controller.items.add(item);

controller.createRunProfile(
  'A: root config',
  vscode.TestRunProfileKind.Run,
  rootHandler,
  true,
  rootTag,
);
controller.createRunProfile(
  'Z: package config',
  vscode.TestRunProfileKind.Run,
  packageHandler,
  true,
  packageTag,
);

Both profiles are compatible and default. Clicking the test's gutter Run button selects A: root config based on its label; there is no independent API through which the provider can prefer Z: package config.

The root configuration and the package configuration are both valid profiles, and both should remain available. The shared TestItem therefore carries both profile tags. The package profile is the better automatic choice for this test because it contains package-specific settings, while the root profile must remain a default so that Run All still includes tests belonging only to the root configuration.

Today, VS Code sorts profiles by isDefault and then by their visible label, then uses the first compatible profile. Consequently, changing a human-readable label can change which configuration executes.

In the Vitest case, the root profile sorts first and the gutter action runs the test with the root configuration. The test then fails because settings defined by the package configuration are absent. Running it explicitly with the package profile succeeds.

Why the existing API is insufficient

• TestRunProfile.tag expresses eligibility, but not preference when a TestItem has several matching tags.
• isDefault is binary and also controls participation in Run All. Marking the root profile non-default would omit root-only tests from Run All.
• Registration order cannot express preference because profiles are subsequently sorted by label.
• Users can select a profile manually, but they first need to understand that an apparently unrelated test failure came from automatic profile selection.
• Encoding priority in leading whitespace or another label prefix couples behavior to presentation. label is user-facing and already has semantic significance for matching Run and Debug profiles.

Proposed API

Add an extension-provided priority used only when VS Code automatically chooses between otherwise equally preferred compatible profiles:

export interface TestRunProfile {
  /**
   * Relative priority used when VS Code automatically chooses between
   * compatible profiles from the same controller and kind.
   *
   * Higher values are preferred. Defaults to 0. A profile selected as a
   * default by the user takes precedence over a non-default profile,
   * regardless of this value.
   */
  priority?: number;
}

Suggested selection order for an automatic single-profile action:

1. Filter by controller, profile kind, and tag compatibility.
2. Prefer profiles currently selected as defaults.
3. Prefer the highest extension-provided priority.
4. Preserve the current label ordering as the tie-breaker.

This should not change explicit profile invocation. If a user chooses a specific profile, that exact profile should run. It should also not change Run All: every selected default profile should continue to participate.

Vitest could then use configuration depth as its preference:

const profile = controller.createRunProfile(
  label,
  vscode.TestRunProfileKind.Run,
  runHandler,
  true,
  configTag,
);

profile.priority = configDepth;

Because tag compatibility is checked first, a high-priority nested profile cannot capture a root-only test. When two configurations at the same depth both include a test, the existing tie-break remains and the user can still select a profile explicitly.

An alternative API shape would be sortText?: string, analogous to TestItem.sortText, with label as the fallback. A numeric priority seems slightly more precise for this use case because it can affect automatic selection without necessarily changing picker presentation.

Backward compatibility

Profiles without a priority would behave exactly as they do today by using priority 0 and retaining label ordering. Existing persisted user-default selections and label-based Run/Debug matching would remain unchanged.

This is not specific to Vitest. Other test providers can have overlapping profiles for nested projects, workspace folders, runtimes, environments, or platforms while still needing one sensible automatic choice for an individual test.

Related issues

• #212628 fixed user-selected defaults being ignored in favor of alphabetical order. This request covers the remaining tie between multiple compatible default profiles.
• #207557 discusses richer profile applicability for workspace folders, but not preference among multiple compatible profiles.