Move driver tests to 1es pool

[Laksh (lakshk98)]

Add 1ES VM-based CI infrastructure for the NTOS and NetEvent eBPF extension driver tests, based on the established ebpf-for-windows test flow.

Key changes:

• Add VM setup, test execution, cleanup, timeout handling, and artifact collection scripts.
• Keep separate CI jobs for ntosebpfext_driver_test.exe and neteventebpfext_driver_test.exe.
• Define extension test suites in PowerShell instead of test_execution.json.
• Register program information for both NTOS and NetEvent extensions during VM setup.
• Add validation for VM mappings and retry transient PowerShell Direct failures.
• Generate kernel dumps on the test VM when a test job times out.
• Add inner-VM ETW tracing for the NTOS and NetEvent extension providers.
• Package the ebpfextensions.wprp profile and collect generated ETL files from the VM.
• Support granular traces around component installation and driver shutdown.
• Improve NetEvent stress-test event draining and avoid treating expected perf-buffer loss as a functional failure.
• Correctly distinguish test failures from job timeouts in the final result.

Closes #370

[Copilot]

Pull request overview

This PR introduces a set of PowerShell modules/scripts and workflow updates to support VM-backed driver test execution (including 1ES runner support), along with packaging the required test payload files into build artifacts.

Changes:

• Add new CI/CD test orchestration scripts/modules for setting up VMs, executing kernel driver tests, collecting logs/dumps/traces, and cleaning up.
• Extend build/test GitHub workflows to support 1ES runner execution and to bundle required VM test payload files (including an eBPF-for-Windows MSI).
• Enhance the existing installer download script with additional parameters and signature validation.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
scripts/vm_run_tests.psm1	Adds helpers to run tests on host vs VM and trigger kernel dumps.
scripts/tracing_utils.psm1	Adds simplified WPR ETW tracing helpers.
scripts/test_execution.json	Adds VM-to-runner mapping and driver test suite definitions.
scripts/setup_ebpf_cicd_tests.ps1	Adds CI/CD setup script to prep VMs/host for driver tests.
scripts/setup_build/setup_build.vcxproj	Adds additional scripts/json into the setup_build copy payload.
scripts/run_driver_tests.psm1	Adds driver test execution wrapper and kernel dump trigger helper.
scripts/Install-eBpfForWindows.ps1	Adds parameters, download-only mode, and Authenticode validation.
scripts/install_ebpf.psm1	Adds MSI-based eBPF component install/stop/uninstall helpers.
scripts/execute_ebpf_cicd_tests.ps1	Adds CI/CD execution script to run selected driver test suites.
scripts/config_test_vm.psm1	Adds VM initialization, artifact export, and result import logic.
scripts/common.psm1	Adds shared logging, timeout/copy helpers, and VM credential utilities.
scripts/cleanup_ebpf_cicd_tests.ps1	Adds CI/CD cleanup script to collect results and stop VMs.
.github/workflows/reusable-test.yml	Updates reusable test workflow for 1ES runner paths and longer timeouts.
.github/workflows/reusable-build.yml	Downloads MSI into build output and validates required VM payload contents.
.github/workflows/cicd.yml	Moves driver tests to 1ES runner execution and adjusts reusable workflow inputs.

Suppressed comments (5)

scripts/execute_ebpf_cicd_tests.ps1:55

• This Import-Module path is not quoted/constructed; $WorkingDirectory\common.psm1 will not expand as a filesystem path and can fail when WorkingDirectory contains spaces. Use Join-Path (or a quoted string) to build the path.

    Import-Module $WorkingDirectory\common.psm1 -Force -ArgumentList ($LogFileName) -WarningAction SilentlyContinue

scripts/execute_ebpf_cicd_tests.ps1:134

• This Import-Module path is not quoted/constructed; $WorkingDirectory\common.psm1 will not expand as a filesystem path reliably. Use Join-Path (or a quoted string) to build the module path.

Import-Module $WorkingDirectory\common.psm1 -Force -ArgumentList ($LogFileName) -ErrorAction SilentlyContinue

scripts/config_test_vm.psm1:322

• Same path construction issue here: $WorkingDirectory\common.psm1 / $WorkingDirectory\install_ebpf.psm1 are not reliably expanded as filesystem paths. Use Join-Path (or a quoted string).

        Import-Module $WorkingDirectory\common.psm1 -ArgumentList ($LogFileName) -Force -WarningAction SilentlyContinue
        Import-Module $WorkingDirectory\install_ebpf.psm1 -ArgumentList ($WorkingDirectory, $LogFileName) -Force -WarningAction SilentlyContinue

scripts/config_test_vm.psm1:347

• This Import-Module uses $WorkingDirectory\common.psm1, which is not reliable path construction (and can fail if the path has spaces). Use Join-Path (or a quoted string) here as well.

            Import-Module $WorkingDirectory\common.psm1 `

scripts/config_test_vm.psm1:350

• This Import-Module uses $WorkingDirectory\install_ebpf.psm1, which is not reliable path construction (and can fail if the path has spaces). Use Join-Path (or a quoted string) here as well.

            Import-Module $WorkingDirectory\install_ebpf.psm1 `

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Suppressed comments (6)

Previously missed (5) — in code that hasn't changed since the last review.

scripts/execute_ebpf_cicd_tests.ps1:66

• When ExecuteOnVM is true, $Config.VMMap.$SelfHostedRunnerName can be $null (missing key) or empty; indexing $VMList[0] will throw with a non-obvious error. Add an explicit validation and error message before using the first VM.

    if ($ExecuteOnVM) {
        Write-Log "Tests will be executed on VM" -ForegroundColor Cyan
        $VMList = $Config.VMMap.$SelfHostedRunnerName
        $VMName = $VMList[0].Name
        $TestWorkingDirectory = "C:\ebpf"

scripts/setup_ebpf_cicd_tests.ps1:82

• When ExecuteOnVM is true, $Config.VMMap.$SelfHostedRunnerName can be missing/empty; later VM initialization and export steps will fail with unclear errors. Validate VMList early and throw a clear message.

    if  ($ExecuteOnVM) {
        $VMList = $Config.VMMap.$SelfHostedRunnerName
        if (-not $VMIsRemote) {
            # Get all local VMs to ready state.
            Initialize-AllVMs -VMList $VMList -ErrorAction Stop

scripts/cleanup_ebpf_cicd_tests.ps1:44

• When ExecuteOnVM is true, $Config.VMMap.$SelfHostedRunnerName can be missing/empty; subsequent cleanup steps will fail and may obscure the real root cause. Validate VMList before using it.

    if ($ExecuteOnVM) {
        $VMList = $Config.VMMap.$SelfHostedRunnerName
        # Wait for all VMs to be in ready state, in case the test run caused any VM to crash.
        Wait-AllVMsToInitialize -VMList $VMList -VMIsRemote $VMIsRemote

.github/workflows/reusable-test.yml:105

• This reusable workflow is triggered via workflow_call, so github.event.workflow_run.head_branch is not defined. Using it for actions/checkout can result in checking out the wrong ref or failing to resolve the ref; use github.sha (or github.ref) to ensure the workflow checks out the exact commit being tested.

    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
      # Check out source code if code coverage is being gathered or if crash dumps
      # (which require ProcDump downloaded via the repo scripts) are being gathered.
      if: ((inputs.code_coverage == true) || (inputs.gather_dumps == true)) && !contains(inputs.environment, '1ES') && (steps.skip_check.outputs.should_skip != 'true')
      with:

scripts/common.psm1:563

• $JobTimedOut is only assigned in the timeout path; when the job completes normally this returns $null, which is easy to mis-handle and makes the return type inconsistent. Initialize it to $false before the loop.

    $TimeElapsed = 0

scripts/config_test_vm.psm1:314

• Typo in log message: "Unnstalling" -> "Uninstalling".

    Write-Log "Unnstalling eBPF components on $VMName"

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/run_driver_tests.psm1:45

• On timeout, the test process is left running. That can leak resources and interfere with subsequent tests / cleanup. The script should terminate the hung process before throwing the TimeoutException.

    if (-not $process.WaitForExit($Timeout * 1000)) {
        Write-Log "$Name exceeded its timeout. A kernel dump will be requested."
        throw [System.TimeoutException]::new("$Name timed out after $Timeout seconds.")
    }

scripts/config_test_vm.psm1:178

• Initialize-AllVMs references $VMIsRemote, but that variable is not defined in module scope and isn’t passed as a parameter. This will always evaluate as $false (or rely on a global variable), which makes the remote-path logic unreliable and easy to break if the function is reused.

    if (-not $VMIsRemote) {
        # Wait for VMs to be ready.
        Write-Log "Waiting for all the VMs to be in ready state..." -ForegroundColor Yellow
        Wait-AllVMsToInitialize -VMList $VMList
    } else {

scripts/config_test_vm.psm1:314

• Typo in log message: "Unnstalling" → "Uninstalling".

    Write-Log "Unnstalling eBPF components on $VMName"

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

scripts/config_test_vm.psm1:166

• Initialize-AllVMs branches on $VMIsRemote, but $VMIsRemote is not defined/passed into the function. This makes the remote-vs-local behavior depend on an outer-scope variable (or default to $false), which can lead to incorrect behavior if the function is ever used for remote VMs.

function Initialize-AllVMs
{
    param ([Parameter(Mandatory=$True)] $VMList)

scripts/config_test_vm.psm1:314

• Typo in log message: "Unnstalling" -> "Uninstalling".

    Write-Log "Unnstalling eBPF components on $VMName"

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

scripts/vm_run_tests.psm1:93

• Stop-eBPFComponents accepts a GranularTracing parameter but never forwards it into the remote script block, so callers toggling GranularTracing currently have no effect. Pass GranularTracing through and call Stop-eBPFServiceAndDrivers with it.

    param([Parameter(Mandatory = $false)][bool] $GranularTracing = $false)

    $scriptBlock = {
        param($WorkingDirectory, $LogFileName)
        Import-Module "$WorkingDirectory\common.psm1" -Force -ArgumentList $LogFileName -WarningAction SilentlyContinue

scripts/setup_ebpf_cicd_tests.ps1:115

• The comment says installing eBPF components "requires psexec" and can’t run in a PowerShell job, but the code path calls Install-eBPFComponents (msiexec-based) inside the job. This comment is misleading and should be removed or corrected.

        # Install eBPF components on host, but skip anything that requires reboot.
        # Note that installing ebpf components requires psexec which does not run in a powershell job.
        Write-Log "Installing eBPF components on host"

.github/workflows/cicd.yml:115

• The 1ES JobId string includes the matrix job-index twice ("${{ strategy.job-index }}" and the "{0}" placeholder). Since reusable-test.yml already injects strategy.job-index via format(..., strategy.job-index), this will produce duplicated suffixes like "...-0-0". Remove one of them (keep "{0}" for formatting).

This issue also appears on line 137 of the same file.

      environment: '["self-hosted", "1ES.Pool=ebpf-extensions-cicd-runner-pool", "1ES.ImageOverride=${{ matrix.image }}_v2", "JobId=netevent-driver-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}-${{ strategy.job-index }}-{0}"]'

.github/workflows/cicd.yml:137

• The 1ES JobId string includes the matrix job-index twice ("${{ strategy.job-index }}" and the "{0}" placeholder). Since reusable-test.yml already injects strategy.job-index via format(..., strategy.job-index), this will produce duplicated suffixes like "...-0-0". Remove one of them (keep "{0}" for formatting).

      environment: '["self-hosted", "1ES.Pool=ebpf-extensions-cicd-runner-pool", "1ES.ImageOverride=${{ matrix.image }}_v2", "JobId=ntos-driver-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}-${{ strategy.job-index }}-{0}"]'

scripts/config_test_vm.psm1:314

• Typo in log message: "Unnstalling" should be "Uninstalling".

    Write-Log "Unnstalling eBPF components on $VMName"

[Copilot]

🟡 Changes recommended

Unresolved CI, VM lifecycle, packaging, tracing, and error-propagation issues remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (12)

.github/workflows/cicd.yml:137

• The two image-matrix jobs generate the same 1ES JobId because this value does not include matrix.image; only the reusable workflow's configuration index ({0}) distinguishes jobs. That can collide in 1ES scheduling and VM mapping, so include the image (or another unique matrix dimension) in the ID.

      environment: '["self-hosted", "1ES.Pool=ebpf-extensions-cicd-runner-pool", "1ES.ImageOverride=${{ matrix.image }}_v2", "JobId=ntos-driver-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}-{0}"]'

.github/workflows/cicd.yml:116

• These driver jobs no longer upload coverage, but .github/codecov.yml:25 still requires after_n_builds: 3. The only remaining code_coverage: true caller is ntosebpfext_units with the default Debug/Release matrix (two uploads), and reusable-test.yml gates uploads on that flag, so Codecov will wait for a build this workflow cannot produce; update the Codecov count with this change.

      code_coverage: false

scripts/config_test_vm.psm1:77

• VMIsRemote is exposed here, but this function always uses local Hyper-V cmdlets such as Get-VMIntegrationService and Enable-VMIntegrationService. The cleanup script passes -VMIsRemote to this function, so remote-VM cleanup will query the runner instead of the VM host and can fail; route remote VMs through the remote readiness/credential path or reject this mode explicitly.

function Wait-AllVMsToInitialize
{
    param([Parameter(Mandatory=$True)]$VMList,
          [Parameter(Mandatory=$false)][bool] $VMIsRemote = $false)

scripts/config_test_vm.psm1:128

• When one VM is still waiting for guest services, already-ready VMs enter this else branch again on the next loop and += attempts to add the same hashtable key twice. With a multi-VM mapping and staggered readiness, this can throw instead of waiting; only add the entry when it is not already present (or assign by key).

            } else {
                Write-Log "Guest services already enabled on $VMName"
                $ReadyList += @{$VMName = $True}

scripts/config_test_vm.psm1:200

• Stop-AllVMs always invokes local Stop-VM and has no VMIsRemote parameter. The cleanup script exposes remote-VM mode, so after remote result collection this call still targets the runner's Hyper-V host and leaves the remote VM running; propagate the mode and stop through the remote management path, or explicitly reject remote cleanup.

function Stop-AllVMs
{
    param ([Parameter(Mandatory=$True)] $VMList)

    foreach ($VM in $VMList) {
        # Stop the VM.
        $VMName = $VM.Name
        Write-Log "Stopping VM $VMName"
        Stop-VM -Name $VMName -Force -TurnOff -WarningAction Ignore  2>&1 | Write-Log
    }

scripts/ebpfextensions.wprp:54

• As in the file profile above, this creates multiple sibling <EventProviders> elements under one collector. WPR expects the provider IDs in the existing block; otherwise the memory profile cannot be started and the requested ETW trace is lost.
  scripts/ebpfextensions.wprp:33
• The provider IDs need to be in the existing single <EventProviders> block for this collector. These sibling <EventProviders> elements make the WPRP profile fail validation when wpr.exe -start loads it, so the new extension tracing will not be collected.
  scripts/install_ebpf.psm1:60
• The NTOS and NetEvent exporters each delete only their own program and section entries, so clearing only $exporterPaths[0] leaves stale NetEvent registrations in a restored VM. Invoke --clear for both exporters before re-registering them.

        Write-Log "Clearing existing eBPF store information"
        & $exporterPaths[0] --clear 2>&1 | Write-Log
        if ($LASTEXITCODE -ne 0) {
            throw "$([System.IO.Path]::GetFileName($exporterPaths[0])) --clear failed with exit code $LASTEXITCODE."
        }

scripts/run_driver_tests.psm1:93

• Both suites hardcode a 1,800-second timeout even though this module exposes $TestHangTimeout and Run-KernelTests passes the caller's value into it. Any caller-provided hang timeout is therefore ignored; use $TestHangTimeout here (or omit -Timeout and let New-TestTuple use its default).

        (New-TestTuple -Suite "ntosebpfext" -Test "ntosebpfext_driver_test.exe" -Timeout 1800),
        (New-TestTuple -Suite "neteventebpfext" -Test "neteventebpfext_driver_test.exe" -Timeout 1800)

scripts/run_driver_tests.psm1:48

• When a driver test exceeds its timeout, this code throws without terminating the test process. The outer job can then fail while the test executable and its driver activity remain running, and the documented dump/cleanup path may not complete reliably. Kill the still-running process and wait for it to exit before throwing the timeout.

    $exited = $process.WaitForExit($Timeout * 1000)
    if (-not $exited -or -not $process.HasExited) {
        Write-Log "$Name exceeded its timeout. A kernel dump will be requested."
        throw [System.TimeoutException]::new("$Name timed out after $Timeout seconds.")

scripts/run_driver_tests.psm1:94

• Both test tuples hard-code a 1,800-second timeout even though the module exposes $TestHangTimeout and callers pass -TestHangTimeout. Any caller selecting a different hang timeout is silently ignored; let New-TestTuple use its default or pass $TestHangTimeout here.

    $testList = @(
        (New-TestTuple -Suite "ntosebpfext" -Test "ntosebpfext_driver_test.exe" -Timeout 1800),
        (New-TestTuple -Suite "neteventebpfext" -Test "neteventebpfext_driver_test.exe" -Timeout 1800)
    )

scripts/setup_build/setup_build.vcxproj:20

• The project now copies ebpfextensions.wprp, but scripts/setup_build/setup_build.vcxproj.filters still references the missing ntosebpfext.wprp. Visual Studio will show a stale/missing project item and the new profile will not be represented in the project filters; update the filters file to match.

    <CopyFileToFolders Include="..\ebpfextensions.wprp">

• Files reviewed: 20/20 changed files
• Comments generated: 9
• Review effort level: Lite