test(vscode): add test-cli lifecycle baseline - #9592
Open
Brian Lam (lambrianmsft) wants to merge 5 commits into
Open
test(vscode): add test-cli lifecycle baseline#9592Brian Lam (lambrianmsft) wants to merge 5 commits into
Brian Lam (lambrianmsft) wants to merge 5 commits into
Conversation
Brian Lam (lambrianmsft)
added
the
risk:medium
Contributor
🤖 AI PR Validation ReportPR Review ResultsThank you for your submission! Here's detailed feedback on your PR title and body compliance:✅ PR Title
✅ Commit Type
✅ Risk Level
✅ What & Why
✅ Impact of Change
✅ Test Plan
✅ Contributors
✅ Screenshots/Videos
Summary Table
All checks pass — this PR title and body are compliant and the declared risk level matches the code impact. Good to merge.Powered by: Copilot CLI (claude-opus-4.8) | Last updated: Tue, 01 Sep 2026 22:28:55 GMT |
Contributor
There was a problem hiding this comment.
Note
Copilot was unable to run its full agentic suite in this review.
Pull request overview
Adds a latest-stable @vscode/test-cli E2E baseline for the VS Code Logic Apps extension, enabling Create Workspace and select lifecycle tests to run independently of ExTester while improving CI visibility.
Changes:
- Added root and app-level
pnpmscripts plus Node wrappers to run label-based CLI E2E and multi-host lifecycle flows. - Introduced new CLI E2E test utilities (dialog guard, CDP helpers, screenshots) and expanded smoke / Create Workspace coverage.
- Updated CI (
vscode-e2e.yml) to run a Create Workspace CLI matrix with structured results + aggregate reporting.
Reviewed changes
Copilot reviewed 33 out of 44 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| package.json | Adds root-level forwarding scripts for new CLI E2E entry points. |
| libs/vscode-extension/src/graphify-out/GRAPH_REPORT.md | Updates generated graph report snapshot metadata/content. |
| libs/data-mapper-v2/src/graphify-out/GRAPH_REPORT.md | Updates generated graph report snapshot metadata/content. |
| libs/chatbot/src/graphify-out/GRAPH_REPORT.md | Updates generated graph report snapshot metadata/content. |
| libs/a2a-core/src/graphify-out/GRAPH_REPORT.md | Updates generated graph report snapshot metadata/content. |
| apps/vs-code-designer/src/test/e2e/workspaceArtifacts.ts | Adds helpers/assertions for generated workspace artifacts (tasks/launch/settings/csproj). |
| apps/vs-code-designer/src/test/e2e/webviewTabs.ts | Adds helpers to find/wait/close webview tabs in VS Code. |
| apps/vs-code-designer/src/test/e2e/visibleDelay.ts | Adds optional “keep host visible” delay for local debug runs. |
| apps/vs-code-designer/src/test/e2e/testUtils.ts | Adds small string/path helper utilities for tests. |
| apps/vs-code-designer/src/test/e2e/screenshot.ts | Adds screenshot capture utilities (Windows screen + CDP). |
| apps/vs-code-designer/src/test/e2e/extension.test.ts | Expands activation smoke to assert dist loading, dependencies, empty startup, and dialog suppression. |
| apps/vs-code-designer/src/test/e2e/dialogGuard.ts | Adds dialog interception/allowlist to fail tests on unexpected prompts. |
| apps/vs-code-designer/src/test/e2e/createWorkspaceTypes.ts | Adds typed models for Create Workspace test cases. |
| apps/vs-code-designer/src/test/e2e/createWorkspaceParityMap.md | Documents ExTester↔CLI parity mapping for Create Workspace + lifecycle coverage. |
| apps/vs-code-designer/src/test/e2e/createWorkspaceCases.ts | Adds curated Create Workspace case generators and label filtering. |
| apps/vs-code-designer/src/test/e2e/commands.test.ts | Replaces generic command listing with explicit Logic Apps command/config assertions + dialog guard. |
| apps/vs-code-designer/src/test/e2e/cdpFormHelpers.ts | Adds CDP-based Create Workspace webview interaction helpers (fields, dropdowns, radios). |
| apps/vs-code-designer/src/test/e2e/cdpClient.ts | Adds a lightweight CDP client to connect to VS Code’s remote debugging targets. |
| apps/vs-code-designer/src/test/e2e/azureAuthWarmup.test.ts | Adds Azure auth warm-up test for durable local profiles. |
| apps/vs-code-designer/src/test/e2e/README.md | Major documentation expansion for running CLI E2E locally and in CI. |
| apps/vs-code-designer/src/main.ts | Adds env-gated minimal activation and env-gated workspace ensure skip for CLI test runs. |
| apps/vs-code-designer/src/app/utils/codeless/getAuthorizationToken.ts | Adds test-gated env token fallback for silent auth; fixes return type to allow undefined. |
| apps/vs-code-designer/src/app/utils/codeless/test/getAuthorizationToken.test.ts | Adds unit coverage for the new test-gated Azure CLI token fallback behavior. |
| apps/vs-code-designer/scripts/summarize-e2e-cli-results.js | Adds Mocha log parsing + JSON/JUnit/summary generation and aggregate reporting. |
| apps/vs-code-designer/scripts/run-e2e-cli.js | Adds wrapper to run labels/lifecycle flows, filter noise, enforce “no dialogs”, and clean temp workspaces. |
| apps/vs-code-designer/scripts/open-e2e-cli-vscode.js | Adds helper to open latest-stable VS Code with an isolated profile for interactive verification/auth. |
| apps/vs-code-designer/package.json | Replaces direct vscode-test call with wrapper scripts for CLI E2E and lifecycle flows. |
| apps/vs-code-designer/.vscode-test.mjs | Expands @vscode/test-cli config to multiple labels, env forwarding, and per-run user-data settings. |
| .squad/knowledge/vscode-e2e-testing.md | Documents learnings/patterns for CLI Create Workspace CI structured results and parity scripts. |
| .github/workflows/vscode-e2e.yml | Adds Create Workspace CLI matrix job + aggregate report job and wires into overall summary. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+545
to
+563
| child.on('close', async (code) => { | ||
| const remainingOutput = outputFilter.flush(); | ||
| if (remainingOutput) { | ||
| process.stdout.write(remainingOutput); | ||
| } | ||
| await cleanupDeferredWorkspaceParent(deferredWorkspaceParent); | ||
|
|
||
| const matchedPattern = forbiddenOutputPatterns.find(({ pattern }) => pattern.test(output)); | ||
| if (matchedPattern) { | ||
| reject(new Error(`\n[activation-smoke] Failed because VS Code output contained: ${matchedPattern.name}`)); | ||
| return; | ||
| } | ||
| if (code && code !== 0) { | ||
| reject(new Error(`Exit code: ${code}`)); | ||
| return; | ||
| } | ||
|
|
||
| resolve(0); | ||
| }); |
Comment on lines
+13
to
+18
| const userDataDirOverride = process.env.LA_E2E_CLI_USER_DATA_DIR; | ||
| const userDataDir = | ||
| userDataDirOverride ?? | ||
| (process.platform === 'win32' | ||
| ? path.join(__dirname, '.vscode-test', userDataSuffix ? `user-data-${userDataSuffix}` : 'user-data') | ||
| : path.join(tmpdir(), `la-vscode-test-${checkoutHash}${userDataSuffix ? `-${userDataSuffix}` : ''}`)); |
Comment on lines
+18
to
+21
| suiteSetup(() => { | ||
| extension = vscode.extensions.getExtension(logicAppsExtensionId); | ||
| activationChannel = vscode.window.createOutputChannel(activationChannelName); | ||
| }); |
Comment on lines
+99
to
+104
| function logActivationEvidence(message: string): void { | ||
| const line = `[activation-smoke] ${message}`; | ||
| console.log(line); | ||
| activationChannel?.appendLine(line); | ||
| activationChannel?.show(true); | ||
| } |
Add latest-stable @vscode/test-cli coverage for activation, command registration, Create Workspace parity labels, generated workspace lifecycle smoke, CI wiring, and documentation while preserving ExTester coverage. Co-authored-by: Copilot App <[email protected]>
Extract shared CLI helper utilities and add an ExTester-to-test-cli Create Workspace parity map for traceability. Co-authored-by: Copilot App <[email protected]>
Add traceability from ExTester Create Workspace cases to @vscode/test-cli labels and extract shared Create Workspace case metadata. Co-authored-by: Copilot App <[email protected]>
Co-authored-by: Copilot App <[email protected]>
Add latest-stable @vscode/test-cli coverage for VS Code activation, workspace creation, generated workspace lifecycles, Azure auth warm-up, MSN Weather designer authoring, and codeful debug task parity. Co-authored-by: Copilot App <[email protected]>
Brian Lam (lambrianmsft)
force-pushed
the
lambrian-microsoft-vs-code-test-cli-baseline
branch
from
September 1, 2026 22:27
ef950fb to
097593f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit Type
Risk Level
What & Why
Adds a latest-stable
@vscode/test-clibaseline for the VS Code Logic Apps extension so selected extension-host smoke and generated-workspace lifecycle tests can run independently of ExTester while preserving ExTester for deep webview DOM coverage. This includes activation and command-registration smoke, Create Workspace parity reporting, Azure auth warm-up helpers, generated Standard/custom-code/rules-engine lifecycle coverage, MSN Weather designer-authored Azure connector coverage, and codeful debug task parity checks.Impact of Change
@vscode/test-cliruns, Azure profile warm-up, MSN Weather lifecycle validation, and CI result summaries.Test Plan
@vscode/test-clihostValidation run locally:
The MSN Weather lifecycle passed with
1 passing (2m), selected the Weather body dynamic-content token, opened both Weather and Response monitoring result panels, and verified the Response body matched the Weather action output.Contributors
Brian Lam (@lambrianmsft)
Screenshots/Videos
N/A - test infrastructure and documentation changes only.