From bd99cc870f77400929047a0e0c686f48b1c3e1a1 Mon Sep 17 00:00:00 2001 From: fadwen <110697945+fadwen@users.noreply.github.com> Date: Wed, 18 Nov 2026 01:13:00 +0000 Subject: [PATCH] chore: sync Copilot and Claude Code standards from standards repo --- .../powershell-standards/powershell-pester.md | 1 + .github/copilot-instructions.md | 12 +- .../pester-supporting-docs/assertion-guide.md | 21 +- .../cicd-integration.md | 928 +++++++++++++++--- .../custom-assertions.md | 14 +- .../integration-test-template.md | 555 ++++++++++- .../mocking-patterns.md | 343 ++++++- .../performance-test-template.md | 467 ++++++++- .../pester-configuration.md | 509 +++++++++- .../security-test-template.md | 652 +++++++++++- .../pester-supporting-docs/test-data-guide.md | 335 ++++++- .../pester-supporting-docs/test-execution.md | 846 +++++++++++++++- .../test-structure-guide.md | 267 ++++- .../unit-test-template.md | 156 ++- .../pester-supporting-docs/v6-migration.md | 144 ++- .../powershell-cicd.instructions.md | 235 ++++- .../powershell-error-handling.instructions.md | 149 ++- .../powershell-functions.instructions.md | 103 +- .../powershell-help.instructions.md | 74 +- .../powershell-modules.instructions.md | 188 +++- .../powershell-pester.instructions.md | 148 ++- .../powershell-platyps.instructions.md | 78 +- .../powershell-pull-requests.instructions.md | 9 +- .../powershell-readme.instructions.md | 153 ++- .../powershell-security.instructions.md | 338 ++++++- .../powershell-style.instructions.md | 88 +- .../powershell-version.instructions.md | 67 +- .../ModuleExample.psm1 | 3 + 28 files changed, 6375 insertions(+), 508 deletions(-) diff --git a/.claude/rules/powershell-standards/powershell-pester.md b/.claude/rules/powershell-standards/powershell-pester.md index 07ff794..93c54df 100644 --- a/.claude/rules/powershell-standards/powershell-pester.md +++ b/.claude/rules/powershell-standards/powershell-pester.md @@ -1,6 +1,7 @@ --- paths: - "**/*.Tests.ps1" + - "**/Tests/**/*.ps1" --- @../../../.github/instructions/powershell-pester.instructions.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 52fce4a..2b39de1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -18,8 +18,7 @@ These are the rules for writing and reviewing PowerShell in this repository. **M ## Topic rules -The rules below are the ones that matter in almost every file. Each topic has a file with the -complete rules, the reasons, and examples: +Each topic has a file with the complete rules, the reasons, and examples: | When the work involves | Read | | --- | --- | @@ -41,6 +40,12 @@ your tooling does not, you MUST read the matching files before creating or chang ## Core rules +These are the rules that matter in almost every file. They are a selection from the topic files, and +each one keeps the ID it has there, so the numbers below are not consecutive: STY-04, for example, is +in the style file. The topic files have gaps of their own, because most of them start each section +at a new ten. A gap does not mean that a rule is missing, and a rule that is only in a topic file +counts as much as one listed here. + ### Version - **VER-01** Write for PowerShell 7.6 (LTS) unless the project declares another target. The @@ -63,7 +68,8 @@ your tooling does not, you MUST read the matching files before creating or chang - **STY-03** Lines SHOULD NOT exceed 115 characters. Shorten a long command with splatting, or break the line after a pipe, comma, or operator. Backtick line continuation MUST NOT be used. - **STY-06** Source MUST be ASCII unless a string literal needs another character: no emoji, - typographic quotes, or en and em dashes in code or comments. + typographic quotes, or en and em dashes in code or comments. Generated documentation and test + output MUST NOT contain emoji either. - **STY-12** Name local variables in descriptive camelCase without a type prefix: `$retryCount`, not `$RetryCount` or `$intRetryCount`. - **STY-13** Automatic variables MUST NOT be assigned or reused as parameter, loop, or test-data diff --git a/.github/instructions/pester-supporting-docs/assertion-guide.md b/.github/instructions/pester-supporting-docs/assertion-guide.md index 47053be..4b6ea1a 100644 --- a/.github/instructions/pester-supporting-docs/assertion-guide.md +++ b/.github/instructions/pester-supporting-docs/assertion-guide.md @@ -17,9 +17,11 @@ Setting `Should.DisableV5 = $true` makes every classic assertion fail with - Each assertion does one job, so its options are its own (`Should-BeString -IgnoreWhitespace`) and its failure message is specific: a caret under the first differing character of a string, a line - diff for a long string, the missing and extra values of a collection. + diff for a long string, the missing and extra values of a collection with the index of each. - `-Expected` decides how the values are compared. `1 | Should-Be $true` and `'1' | Should-Be 1` both pass, so assert the type as well when the type matters. +- `$null`, an empty collection, and a single-item array are handled by the same rules in every + assertion of a family. The table under "Pipeline Or -Actual" lists them. - A value assertion refuses a collection as `-Expected`. `Should-Be -Expected @(1, 2)` is an error that names `Should-BeCollection`. - The set can be extended. See the [custom assertion guide](./custom-assertions.md). @@ -72,7 +74,7 @@ the pipeline did and points to `-Actual`. | `Should -BeGreaterOrEqual $x` | `Should-BeGreaterThanOrEqual $x` | Name changed | | `Should -BeLessOrEqual $x` | `Should-BeLessThanOrEqual $x` | Name changed | | `Should -Contain $item` | `Should-ContainCollection $item` | | -| `$item \| Should -BeIn $list` | `$list \| Should-ContainCollection $item` | The list is piped | +| `$item \| Should -BeIn $list` | `$list \| Should-ContainCollection $item` | The list is piped. `$list \| Should-Any { $_ -eq $item }` also works | | `Should -HaveCount 3` | `Should-BeCollection -Count 3` | | | `Should -Throw '*msg*'` | `Should-Throw -ExceptionMessage '*msg*'` | Both need the wildcards | | `Should -Invoke Cmd` | `Should-Invoke Cmd` | | @@ -92,6 +94,10 @@ $result.Items | Should-BeFalsy `Should-BeFalsy` passes for `$null`, `''`, `0`, `$false`, and `@()`. `'' | Should-BeNull` fails, and so does `$null | Should-BeEmptyString`. +A file that keeps the classic style keeps `Should -Exist`. Pester 6 gave it a `-LiteralPath` switch: +`$path | Should -Exist -LiteralPath` passes for a path that contains brackets, where the plain form +reads the brackets as a wildcard and fails. + ## The Four Families | Family | Examples | Use for | @@ -131,9 +137,10 @@ Get-Content -LiteralPath $reportPath -Raw | Should-BeString $expected -Normalize #### Long Strings -A failing `Should-BeString` prints a line diff in place of the caret when the strings have more than -one line and either more than 10 lines or more than 120 characters. Expected line numbers are on the -left and actual on the right: +From 6.2, a failing `Should-BeString` prints a line diff in place of the caret when the strings have +more than one line and either more than 10 lines or more than 120 characters. Pester 6.1 prints the +caret for the same strings. Expected line numbers are on the left and actual on the right, so the +output stays readable when lines were added or removed: ```text Expected strings to be the same, but they were different. @@ -150,6 +157,9 @@ Expected 11 line(s), actual 11 line(s). 7 7 | line 7 of the expected text ``` +Only the differing lines are expanded. In them a tab or a trailing space is drawn as a visible +symbol, and the unchanged lines around them are printed as they are, not turned into escape codes. + This makes `Should-BeString` the assertion for rendered templates and generated files. `Should-Be` on the same strings prints both in full. @@ -419,6 +429,7 @@ that later lines depend on, such as a count before an index. - Assert the narrowest fact that proves the behaviour. `Should-Be 'Active'` is better than `Should-NotBeNull` (TST-11). - Assert the count before reading an element by index (TST-12). +- Use `Should-HaveType` rather than asserting on `.GetType().Name` as a string (TST-13). - Prefer one `Should-BeEquivalent -ExcludePathsNotOnExpected` to a run of property assertions, and add index assertions when order matters. - Assert an error by its type or error ID. A message that the code owns changes more often. diff --git a/.github/instructions/pester-supporting-docs/cicd-integration.md b/.github/instructions/pester-supporting-docs/cicd-integration.md index a661e43..ce8b586 100644 --- a/.github/instructions/pester-supporting-docs/cicd-integration.md +++ b/.github/instructions/pester-supporting-docs/cicd-integration.md @@ -4,9 +4,13 @@ This guide shows how to run Pester in a pipeline so that the build fails when it the CI rules in [powershell-cicd.instructions.md](../powershell-cicd.instructions.md) and rules TST-40 to TST-42 in [powershell-pester.instructions.md](../powershell-pester.instructions.md). -The gate script was run on Pester 6.2.0 under PowerShell 7.6 and Windows PowerShell 5.1. The -pipeline definitions are well-formed, and the GitHub Actions and Azure Pipelines keywords were -checked against their documentation, but none of the pipelines was run on a CI service. +The gate script and the runner script were run on Pester 6.2.0 under PowerShell 7.6 and Windows +PowerShell 5.1, and the inline scripts of the GitHub Actions workflow were run locally. The pipeline +definitions are well-formed, but none of the pipelines was run on a CI service, and the keywords of +the restored jobs were not checked again against the documentation of the services. + +Do not use emoji in generated code, in documentation, or in test output. Use plain text and ASCII +characters only. ## What the Pipeline Must Do @@ -30,10 +34,47 @@ Three facts about Pester 6.2.0 shape the gate: returns nothing. The gate therefore tests for `$null` and for `TotalCount`. 3. `CodeCoverage.CoveragePercentTarget` does not fail a run. The script compares the percentage. +## Job Design + +Three more facts shape the jobs: + +1. Coverage and parallel runs pull against each other. Pester 6.1 and later merge coverage across + the workers of a parallel run, but collect it with breakpoints to do so. Run a fast parallel job + without coverage for feedback, and a separate sequential job with the profiler for the gate. +2. A file that fails discovery adds nothing to `FailedCount`. Every gate checks `Result`, and a + cheap discovery-only job runs first. +3. Pester 6.0 and 6.1 do not import on PowerShell 7.4.0 to 7.4.5. Their `net8.0` assembly referred + to a newer `System.Management.Automation` than those releases carry, so `Import-Module` failed + with `Cannot convert "PesterConfigurationDeserializer" from String to Type`. Pester 6.2 fixed it. + Hosted runners carry the latest patch release, so this bites self-hosted agents and pinned + container images; the pin on Pester 6.2.0 (CIC-02) avoids it. + +```text +validate (discovery only, fast) + | + +--> test (operating system matrix, parallel, no coverage: fast feedback) + +--> coverage (sequential, coverage gate) + +--> test-windows-powershell (Windows PowerShell 5.1, when the project declares it) +``` + ## The Gate Script -Keep the logic in one script that the pipeline and a developer both call. The test folder is held in -one place, the `TestRoot` parameter. +Keep the logic in one script, `Tests/Invoke-Tests.ps1`, that the pipeline and a developer both call. +The standard one is the runner in +[Test Execution Guide](./test-execution.md#the-test-runner-script): it takes the test type and the +environment, writes the reports, and applies the gate. The pipelines in this guide call it. CI +passes `-Environment CI`, and `-CodeCoverage` in the job that holds the coverage gate; without that +switch the runner does not measure coverage. + +The script and the settings files are kept in the `Tests` folder, as the layout in +[Pester Test Structure Guide](./test-structure-guide.md#test-directory-layout) shows. A `build` +folder beside `Tests` is the alternative; both are one level below the repository root, so only the +path in each call changes. + +The alternative is the minimal script below: only the gate, with its reports. With it, each call +in a pipeline becomes `./Tests/Invoke-Tests.ps1` without the test type and environment, and the +reports are `out/testResults.xml` and `out/coverage.xml`. The test folder is held in one place, the +`TestRoot` parameter. ```powershell #requires -Version 7.6 @@ -61,12 +102,12 @@ Format of the test result report. Format of the coverage report. .EXAMPLE -./build/Invoke-Tests.ps1 +./Tests/Invoke-Tests.ps1 Runs the tests of the repository and applies the gate. .EXAMPLE -./build/Invoke-Tests.ps1 -TestResultFormat JUnitXml -CoverageFormat Cobertura +./Tests/Invoke-Tests.ps1 -TestResultFormat JUnitXml -CoverageFormat Cobertura Writes the reports in the formats that GitLab CI reads. @@ -113,6 +154,7 @@ $configuration.CodeCoverage.Path = @( Join-Path -Path $repositoryRoot -ChildPath 'Public' Join-Path -Path $repositoryRoot -ChildPath 'Private' ) +$configuration.CodeCoverage.CoveragePercentTarget = 80 $configuration.CodeCoverage.OutputFormat = $CoverageFormat $configuration.CodeCoverage.OutputPath = Join-Path -Path $OutputPath -ChildPath 'coverage.xml' $configuration.TestResult.Enabled = $true @@ -133,15 +175,20 @@ if ($result.Result -ne 'Passed') { } $coverage = [math]::Round($result.CodeCoverage.CoveragePercent, 2) -if ($coverage -lt 80) { - throw "Code coverage is $coverage percent. The minimum is 80." +$target = $result.CodeCoverage.CoveragePercentTarget +if ($coverage -lt $target) { + foreach ($file in $result.CodeCoverage.CommandsMissed | Group-Object -Property File) { + Write-Host " $($file.Name): $($file.Count) uncovered command(s)" + } + throw "Code coverage is $coverage percent. The minimum is $target." } ``` Notes on the script: - It is the gate from the testing rules, with reports and parameters added. The three checks are - the same, in the same order. + the same, in the same order. The coverage check compares with the configured target, which the + script sets to 80 (TST-40), and lists the uncovered commands for each file when it fails. - Every path comes from `$PSScriptRoot`, so the script works from any current location. - `Run.RepoRoot` is set, so shared setup files are found even in a checkout that has no `.git` folder; see [Pester Configuration Guide](./pester-configuration.md#runreporoot). @@ -158,6 +205,320 @@ To enforce tags in CI (TST-04), add the check for untagged tests from ## GitHub Actions +```yaml +name: PowerShell Testing + +on: + push: + branches: [main, develop] + pull_request: + branches: [main] + schedule: + # Every night at 02:00 UTC + - cron: '0 2 * * *' + +permissions: + contents: read + +env: + POWERSHELL_TELEMETRY_OPTOUT: 1 + PESTER_VERSION: '6.2.0' + +jobs: + # Fast structural check. It finds an empty -ForEach, a file that cannot be discovered on its + # own, and a configuration value of the wrong type, without running a test. + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + + - name: Install Pester + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + Install-PSResource -Name Pester -Version $env:PESTER_VERSION -TrustRepository + + - name: Discovery-only pass + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + Import-Module -Name Pester -RequiredVersion $env:PESTER_VERSION + + $configuration = New-PesterConfiguration + $configuration.Run.Path = Join-Path -Path $env:GITHUB_WORKSPACE -ChildPath 'Tests' + $configuration.Run.RepoRoot = $env:GITHUB_WORKSPACE + $configuration.Run.SkipRun = $true + $configuration.Run.PassThru = $true + $configuration.Output.CIFormat = 'GithubActions' + + $result = Invoke-Pester -Configuration $configuration + + if ($null -eq $result) { + throw 'Pester found no test files.' + } + if ($result.Result -ne 'Passed') { + foreach ($container in $result.FailedContainers) { + Write-Host "::error file=$($container.Item)::Discovery failed" + } + throw "$($result.FailedContainersCount) file(s) failed discovery." + } + Write-Host "Discovered $($result.TotalCount) tests in $(@($result.Containers).Count) files." + + - name: Verify every test is tagged + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + Import-Module -Name Pester -RequiredVersion $env:PESTER_VERSION + + $configuration = New-PesterConfiguration + $configuration.Run.Path = Join-Path -Path $env:GITHUB_WORKSPACE -ChildPath 'Tests' + $configuration.Run.RepoRoot = $env:GITHUB_WORKSPACE + $configuration.Run.SkipRun = $true + $configuration.Run.PassThru = $true + # The reserved value None selects the tests that have no tag. + $configuration.Filter.Tag = 'None' + $configuration.Output.Verbosity = 'None' + + $result = Invoke-Pester -Configuration $configuration + + # ShouldRun marks what the filter selected. TotalCount ignores the filter. + $untagged = @($result.Tests | Where-Object -Property ShouldRun) + foreach ($test in $untagged) { + Write-Host "::error::Untagged test: $($test.ExpandedPath)" + } + if ($untagged.Count -gt 0) { + throw "$($untagged.Count) test(s) have no tag." + } + + test: + needs: validate + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + # Hosted runners carry a supported PowerShell 7. Do not add a PowerShell version that is + # past its end-of-support date (CIC-04). + os: [windows-latest, ubuntu-latest, macos-latest] + steps: + - uses: actions/checkout@v7 + + - name: Cache PowerShell modules + uses: actions/cache@v6 + with: + path: | + ~/.local/share/powershell/Modules + ~/Documents/PowerShell/Modules + ~/Documents/WindowsPowerShell/Modules + key: ${{ runner.os }}-pester-${{ env.PESTER_VERSION }}-${{ hashFiles('**/*.psd1') }} + restore-keys: | + ${{ runner.os }}-pester-${{ env.PESTER_VERSION }}- + + - name: Install build tools + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + Install-PSResource -Name PSScriptAnalyzer -Version 1.25.0 -TrustRepository + Install-PSResource -Name Pester -Version $env:PESTER_VERSION -TrustRepository + + - name: Run PSScriptAnalyzer + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + $findings = @(Invoke-ScriptAnalyzer -Path . -Recurse -Settings PSGallery) + $findings | Format-Table -Property Severity, RuleName, ScriptName, Line -AutoSize + if ($findings.Count -gt 0) { + throw "PSScriptAnalyzer reported $($findings.Count) finding(s)." + } + + # Fast feedback: the unit tests in parallel, without coverage. + - name: Run unit tests + shell: pwsh + run: ./Tests/Invoke-Tests.ps1 -TestType Unit -Environment CI -Parallel + + # Integration tests share external resources, so they never run in parallel. + - name: Run integration tests + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + $result = ./Tests/Invoke-Tests.ps1 -TestType Integration -Environment CI -PassThru + if ($result.Result -ne 'Passed') { + # A failed integration test is reported as a warning and does not fail the build. + Write-Warning "Integration tests failed: $($result.FailedCount) failure(s)." + } + # Invoke-Pester left the number of failures in the exit code, which fails the step. + $global:LASTEXITCODE = 0 + + - name: Upload test results + if: always() + uses: actions/upload-artifact@v7 + with: + name: test-results-${{ matrix.os }} + path: Tests/Results/ + + # Coverage runs sequentially, so it can use the profiler-based tracer. Like every run of the + # runner it is shuffled: a new seed for each run, printed at the start and named when it fails. + coverage: + needs: validate + runs-on: windows-latest + steps: + - uses: actions/checkout@v7 + + - name: Install Pester + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + Install-PSResource -Name Pester -Version $env:PESTER_VERSION -TrustRepository + + - name: Run tests with coverage + shell: pwsh + run: ./Tests/Invoke-Tests.ps1 -TestType Unit -Environment CI -CodeCoverage + + - name: Upload coverage + if: always() + uses: actions/upload-artifact@v7 + with: + name: coverage + path: Tests/Results/ + + - name: Upload coverage to Codecov + uses: codecov/codecov-action@v7 + with: + files: Tests/Results/Coverage-*.xml + flags: powershell + env: + CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + + security-scan: + needs: validate + runs-on: windows-latest + steps: + - uses: actions/checkout@v7 + + - name: Install Pester + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + Install-PSResource -Name Pester -Version $env:PESTER_VERSION -TrustRepository + + # A security suite that discovers no tests must fail the build: the runner throws + # 'Pester ran no tests.' Attack data built in BeforeAll instead of BeforeDiscovery empties + # the suite without any other message. + - name: Run security tests + shell: pwsh + run: ./Tests/Invoke-Tests.ps1 -TestType Security -Environment CI + + - name: Run credential scan + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + $credentialPatterns = @( + '(?i)(password|pwd|pass)\s*[:=]\s*["''][^"''\s]{8,}' + '(?i)(api[_-]?key|apikey)\s*[:=]\s*["'']?[a-zA-Z0-9]{20,}' + '(?i)(secret|token)\s*[:=]\s*["'']?[a-zA-Z0-9]{16,}' + ) + + $files = Get-ChildItem -Path . -Recurse -File -Include '*.ps1', '*.psm1', '*.psd1' + $leaks = @( + foreach ($file in $files) { + Select-String -LiteralPath $file.FullName -Pattern $credentialPatterns + } + ) + + # Only the place is printed. The matched text may be a secret (CIC-06). + foreach ($leak in $leaks) { + Write-Warning "Potential credential leak in $($leak.Path), line $($leak.LineNumber)" + } + if ($leaks.Count -gt 0) { + throw "Credential leaks detected: $($leaks.Count) issue(s)." + } + + performance-baseline: + needs: validate + runs-on: windows-latest + steps: + - uses: actions/checkout@v7 + + - name: Install Pester + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + Install-PSResource -Name Pester -Version $env:PESTER_VERSION -TrustRepository + + # Never in parallel and never with coverage: both distort the measurement. + - name: Run performance tests + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + $result = ./Tests/Invoke-Tests.ps1 -TestType Performance -Environment CI -PassThru + if ($result.Result -ne 'Passed') { + # Performance tests give warnings, not failures. + Write-Warning "Performance regression: $($result.FailedCount) test(s) exceeded the baseline." + } + $global:LASTEXITCODE = 0 + + # The folder holds the test results and the metrics file of the run, for trending. + - name: Upload performance results + if: always() + uses: actions/upload-artifact@v7 + with: + name: performance-results + path: Tests/Results/ +``` + +### GitHub Actions Notes + +- `actions/setup-powershell` is not an official action. GitHub-hosted runners have a supported + PowerShell 7 installed (`shell: pwsh`), and `windows-latest` also has Windows PowerShell 5.1 + (`shell: powershell`). Select the shell instead of installing a version. +- When a step needs a specific release, assert it instead of assuming the runner image. The runner + script and the gate script do so with their `#requires -Version 7.6` line; an inline step does it + with a test: + + ```powershell + if ($PSVersionTable.PSVersion -lt [version]'7.6') { + throw "PowerShell 7.6 or later is needed, and this is $($PSVersionTable.PSVersion)." + } + ``` + +- `::set-output` is deprecated: the runner warns when a step uses it. Write step outputs to the file that + `GITHUB_OUTPUT` names, as the runner does: + `Add-Content -LiteralPath $env:GITHUB_OUTPUT -Value 'name=value'`. The runner also writes a table + of the counts, and the coverage figure, to `GITHUB_STEP_SUMMARY`. +- `Output.CIFormat = 'GithubActions'` makes Pester write `::error` and `::warning` annotations, + which show a failure inline on the pull request diff. `Auto` detects GitHub Actions, but setting + the value explicitly is clearer; `Tests/PesterConfiguration.CI.psd1` sets it. The alternative is + to set nothing and rely on `Auto`. +- `fail-fast: false` keeps the failure of one platform from cancelling the others, so that the run + shows every platform that broke. +- The cache key holds the Pester version and the hash of the `.psd1` files, so a new pin or a + changed manifest fills the cache again. +- The matrix covers Windows, Linux, and macOS (CIC-16). A project that supports one operating + system keeps one job; see [Alternative: One Job](#alternative-one-job). +- The nightly schedule runs the same jobs without a commit (CIC-17), which finds a failure that + comes from the runner image or from a dependency. +- The PSScriptAnalyzer step uses the `PSGallery` preset, the rules that the PowerShell Gallery + applies. The alternatives are `-Severity Error, Warning` and the settings file of the repository; + see [powershell-cicd.instructions.md](../powershell-cicd.instructions.md). +- The Codecov token comes from a repository secret (CIC-06). Codecov reads `JaCoCo` and + `Cobertura` reports. +- The credential scan is part of the security gate (CIC-12). It prints the file and the line of a + match, never the matched text. +- The integration and performance steps report a failure as a warning; see + [Failing Correctly](#failing-correctly). +- The security and performance jobs (CIC-12, CIC-13) run the tests that TST-43 requires: security + tests for credential handling and input validation, and performance tests for a function that + processes more than 100 items. Remove a job whose test folder does not exist: a run that finds + no test file fails. +- `Run.Parallel` is experimental. The `-Parallel` switch of the unit test step is the owner's + default for fast feedback; without the switch the step runs the same tests sequentially. +- Each action is referenced by a release tag (CIC-09). A full commit ID is the stricter choice; look + the ID up and write the tag after it in a comment. + +### Alternative: One Job + +A project that supports one operating system, and that uses the minimal gate script, needs one job +that runs on pull requests: + ```yaml name: Quality gates @@ -193,7 +554,7 @@ jobs: - name: Run Pester shell: pwsh - run: ./build/Invoke-Tests.ps1 + run: ./Tests/Invoke-Tests.ps1 - name: Upload reports if: always() @@ -205,9 +566,6 @@ jobs: Use `runs-on: windows-latest` when the code is Windows-only. -Each action is referenced by a release tag (CIC-09). A full commit ID is the stricter choice; look -the ID up and write the tag after it in a comment. - The gate script requires PowerShell 7.6 through its `#requires` line, so the job fails with a clear message on a runner image that has an older version. @@ -217,7 +575,8 @@ shows as an annotation on the run. No option has to be set for this. ### Windows PowerShell 5.1 Job Add this job only when the project declares Windows PowerShell 5.1 support (CIC-03). It goes under -`jobs:` beside the `test` job. +`jobs:` beside the `test` job. From Pester 6.2 a parallel run works under Windows PowerShell 5.1 as +well. ```yaml test-windows-powershell: @@ -233,147 +592,456 @@ Add this job only when the project declares Windows PowerShell 5.1 support (CIC- - name: Run Pester shell: powershell - run: ./build/Invoke-Tests.ps1 + run: ./Tests/Invoke-Tests.ps1 ``` +With the runner script, the last step calls it with `-TestType Unit -Environment CI`. With either +script, the first line of the script is then `#requires -Version 5.1`. + `Install-Module` is used here because PSResourceGet is not part of Windows PowerShell 5.1 (VER-08). Windows PowerShell ships with Pester 3.4.0, which has a different publisher, so the installation -needs `-SkipPublisherCheck`. +needs `-SkipPublisherCheck`. A step that runs under both versions tests for the command first: + +```powershell +# Install-PSResource is part of PowerShell 7.4 and later. Windows PowerShell 5.1 needs Install-Module. +if (Get-Command -Name Install-PSResource -ErrorAction Ignore) { + Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository +} else { + Set-PSRepository -Name PSGallery -InstallationPolicy Trusted + Install-Module -Name Pester -RequiredVersion 6.2.0 -Scope CurrentUser -Force -SkipPublisherCheck +} +``` ## Azure Pipelines ```yaml +# azure-pipelines.yml trigger: branches: include: - main + - develop + paths: + exclude: + - README.md + - docs/* pr: branches: include: - main - -pool: - vmImage: ubuntu-latest - -steps: - - task: PowerShell@2 - displayName: Install build tools - inputs: - targetType: inline - pwsh: true - script: | - $ErrorActionPreference = 'Stop' - Install-PSResource -Name PSScriptAnalyzer -Version 1.25.0 -TrustRepository - Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository - - - task: PowerShell@2 - displayName: Run PSScriptAnalyzer - inputs: - targetType: inline - pwsh: true - script: | - $ErrorActionPreference = 'Stop' - $findings = @(Invoke-ScriptAnalyzer -Path . -Recurse -Severity Error, Warning) - $findings | Format-Table -Property Severity, RuleName, ScriptName, Line -AutoSize - if ($findings.Count -gt 0) { - throw "PSScriptAnalyzer reported $($findings.Count) finding(s)." - } - - - task: PowerShell@2 - displayName: Run Pester - inputs: - targetType: filePath - filePath: build/Invoke-Tests.ps1 - pwsh: true - - - task: PublishTestResults@2 - displayName: Publish test results - condition: always() - inputs: - testResultsFormat: NUnit - testResultsFiles: out/testResults.xml - - - task: PublishCodeCoverageResults@2 - displayName: Publish code coverage - condition: always() - inputs: - summaryFileLocation: out/coverage.xml + paths: + exclude: + - README.md + - docs/* + +schedules: + - cron: '0 2 * * *' + displayName: Daily build + branches: + include: + - main + +variables: + POWERSHELL_TELEMETRY_OPTOUT: 1 + +stages: + - stage: Test + displayName: Testing stage + jobs: + - job: TestWindows + displayName: Test on Windows + pool: + vmImage: windows-latest + strategy: + matrix: + PS7: + powershellVersion: '7' + usePwsh: true + # Keep this leg only when the project declares Windows PowerShell 5.1 support (CIC-03). + PS5: + powershellVersion: '5.1' + usePwsh: false + steps: + - task: PowerShell@2 + displayName: Install build tools + inputs: + targetType: inline + pwsh: $(usePwsh) + script: | + $ErrorActionPreference = 'Stop' + Set-PSRepository -Name PSGallery -InstallationPolicy Trusted + Install-Module -Name Pester -RequiredVersion 6.2.0 -Scope CurrentUser -Force -SkipPublisherCheck + Install-Module -Name PSScriptAnalyzer -RequiredVersion 1.25.0 -Scope CurrentUser -Force + + - task: PowerShell@2 + displayName: Run PSScriptAnalyzer + inputs: + targetType: inline + pwsh: $(usePwsh) + script: | + $ErrorActionPreference = 'Stop' + $findings = @(Invoke-ScriptAnalyzer -Path . -Recurse -Settings PSGallery) + $findings | Format-Table -Property Severity, RuleName, ScriptName, Line -AutoSize + if ($findings.Count -gt 0) { + Write-Host "##vso[task.logissue type=error]PSScriptAnalyzer found $($findings.Count) issue(s)" + throw "PSScriptAnalyzer reported $($findings.Count) finding(s)." + } + + # The runner sets the variables TotalTests, PassedTests, FailedTests, and CodeCoverage + # with ##vso[task.setvariable]. + - task: PowerShell@2 + displayName: Run tests + inputs: + targetType: filePath + filePath: Tests/Invoke-Tests.ps1 + arguments: -TestType All -Environment CI -CodeCoverage + pwsh: $(usePwsh) + + - task: PublishTestResults@2 + displayName: Publish test results + condition: always() + inputs: + testResultsFormat: NUnit + testResultsFiles: Tests/Results/TestResults*.xml + testRunTitle: PowerShell Tests - $(Agent.OS) - PS$(powershellVersion) + + # PublishCodeCoverageResults@1 is deprecated. Version 2 takes summaryFileLocation + # directly and infers the format. + - task: PublishCodeCoverageResults@2 + displayName: Publish code coverage + condition: always() + inputs: + summaryFileLocation: Tests/Results/Coverage*.xml + + - job: TestLinux + displayName: Test on Linux + pool: + vmImage: ubuntu-latest + steps: + - task: PowerShell@2 + displayName: Install Pester + inputs: + targetType: inline + pwsh: true + script: | + $ErrorActionPreference = 'Stop' + Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository + + # Only the unit tests that are tagged CrossPlatform run on Linux. + - task: PowerShell@2 + displayName: Run cross-platform tests + inputs: + targetType: filePath + filePath: Tests/Invoke-Tests.ps1 + arguments: -TestType Unit -Tag CrossPlatform -Environment CI + pwsh: true + + - stage: Security + displayName: Security scanning + dependsOn: Test + condition: succeeded() + jobs: + - job: SecurityScan + displayName: Security analysis + pool: + vmImage: windows-latest + steps: + - task: PowerShell@2 + displayName: Install Pester + inputs: + targetType: inline + pwsh: true + script: | + $ErrorActionPreference = 'Stop' + Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository + + - task: PowerShell@2 + displayName: Security tests + inputs: + targetType: filePath + filePath: Tests/Invoke-Tests.ps1 + arguments: -TestType Security -Environment CI + pwsh: true + + - stage: Deploy + displayName: Deployment + dependsOn: + - Test + - Security + condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main')) + jobs: + - deployment: DeployProduction + displayName: Deploy to PowerShell Gallery + environment: Production + strategy: + runOnce: + deploy: + steps: + - task: PowerShell@2 + displayName: Publish to PowerShell Gallery + inputs: + targetType: inline + pwsh: true + script: | + # Publish logic here + Write-Host 'Publishing to PowerShell Gallery...' ``` -`pwsh: true` runs the step in PowerShell 7. On a Windows agent, `pwsh: false` runs it in Windows -PowerShell 5.1; use that only in a job for a project that declares 5.1 support. - -Pester detects Azure Pipelines and writes failures as logging commands. No option has to be set for -this. +Notes on the pipeline: + +- The `PS5` leg is for a project that declares Windows PowerShell 5.1 support (CIC-03); remove it + otherwise. `pwsh: true` runs a step in PowerShell 7, and on a Windows agent `pwsh: false` runs + it in Windows PowerShell 5.1. With that leg, the first line of the runner is + `#requires -Version 5.1`. +- The Windows job installs with `Set-PSRepository` and `Install-Module`, which work in both legs. + A job that runs only PowerShell 7, such as the Linux job, uses `Install-PSResource` (VER-08). +- The path filters keep a change that touches only `README.md` or `docs/` from starting a build. +- `PublishTestResults@2` reads the NUnit report, and both publish tasks run with + `condition: always()` so that a failed run still shows its results. +- Pester detects Azure Pipelines and writes failures as logging commands. The CI configuration + file can set `Output.CIFormat = 'AzureDevops'` to state it; no option has to be set. +- Tag the tests that run on every platform `CrossPlatform`. `-Tag` replaces the tag of the test + type, so the Linux job runs the tests under `Tests/Unit` that have that tag. +- The deployment stage runs after the test and security stages, on the main branch. The rule for + publishing a module is narrower: CIC-08 publishes only from a tagged release, with the API key + from a secret. To follow it here, make the condition + `and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/tags/v'))` and add the tag + to the trigger. ## Other CI Systems -The pattern is the same everywhere: install the pinned tools, call the gate script, and publish the +The pattern is the same everywhere: install the pinned tools, call the script, and publish the reports that the script wrote. A step fails when the script throws. -GitLab CI reads JUnit test results and Cobertura coverage: - -```yaml -test: - stage: test - # An image that contains PowerShell 7.6. - image: registry.example.com/build/powershell:7.6 - script: - - pwsh -NoProfile -Command "Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository" - - pwsh -NoProfile -File ./build/Invoke-Tests.ps1 -TestResultFormat JUnitXml -CoverageFormat Cobertura - artifacts: - when: always - reports: - junit: out/testResults.xml - coverage_report: - coverage_format: cobertura - path: out/coverage.xml -``` +### Jenkins -Jenkins reads JUnit test results with the `junit` step: +Jenkins reads JUnit test results with the `junit` step, so set +`TestResult.OutputFormat = 'JUnitXml'` in `Tests/PesterConfiguration.CI.psd1`. ```groovy pipeline { - agent { label 'pwsh' } + agent none + + environment { + POWERSHELL_TELEMETRY_OPTOUT = '1' + } stages { stage('Test') { + parallel { + // Keep this stage only when the project declares Windows PowerShell 5.1 support. + stage('Windows PowerShell 5.1') { + agent { label 'windows' } + steps { + powershell ''' + $ErrorActionPreference = 'Stop' + Set-PSRepository -Name PSGallery -InstallationPolicy Trusted + Install-Module -Name Pester -RequiredVersion 6.2.0 -Scope CurrentUser -Force -SkipPublisherCheck + ./Tests/Invoke-Tests.ps1 -TestType All -Environment CI + ''' + } + post { + always { + junit 'Tests/Results/TestResults*.xml' + } + } + } + + stage('PowerShell 7') { + agent { label 'pwsh' } + steps { + pwsh ''' + $ErrorActionPreference = 'Stop' + Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository + ./Tests/Invoke-Tests.ps1 -TestType All -Environment CI -CodeCoverage + ''' + } + post { + always { + junit 'Tests/Results/TestResults*.xml' + publishCoverage adapters: [jacocoAdapter('Tests/Results/Coverage*.xml')] + } + } + } + } + } + + stage('Security Scan') { + agent { label 'pwsh' } + steps { + pwsh './Tests/Invoke-Tests.ps1 -TestType Security -Environment CI' + } + } + + stage('Deploy') { + when { + branch 'main' + } + agent { label 'pwsh' } steps { - pwsh 'Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository' - pwsh './build/Invoke-Tests.ps1 -TestResultFormat JUnitXml' + pwsh ''' + # Deployment logic + Write-Host 'Deploying to PowerShell Gallery...' + ''' } } } post { always { - junit 'out/testResults.xml' + emailext( + to: '${DEFAULT_RECIPIENTS}', + subject: '${PROJECT_NAME} - Build ${BUILD_NUMBER} - ${BUILD_STATUS}', + body: '''${PROJECT_NAME} - Build ${BUILD_NUMBER} - ${BUILD_STATUS} + +Test Results: +Total Tests: ${TEST_COUNTS,var="total"} +Failed Tests: ${TEST_COUNTS,var="fail"} +Passed Tests: ${TEST_COUNTS,var="pass"} +Skipped Tests: ${TEST_COUNTS,var="skip"} + +Build URL: ${BUILD_URL} +''' + ) } } } ``` -For a container image, Microsoft's documentation states that the .NET SDK images contain PowerShell -and that they are meant for development and testing. Check which PowerShell version an image -contains before you rely on it; the `#requires` line of the gate script fails the job when the -version is too old. +The `post` block mails a summary of the build and of the test counts after every run. The coverage +step publishes the JaCoCo report; which step does this depends on the coverage plugin that is +installed. + +### GitLab CI + +```yaml +stages: + - test + - security + - deploy + +variables: + POWERSHELL_TELEMETRY_OPTOUT: "1" + +.powershell_template: &powershell_template + before_script: + - pwsh -NoProfile -Command "Install-PSResource -Name Pester -Version 6.2.0 -TrustRepository" + +# GitLab needs JUnit for test results and Cobertura for coverage. Pester 6 writes both: set +# TestResult.OutputFormat = 'JUnitXml' and CodeCoverage.OutputFormat = 'Cobertura' in +# Tests/PesterConfiguration.CI.psd1. +# +# Container images: the mcr.microsoft.com/powershell images are deprecated. PowerShell now ships +# inside the .NET SDK images: mcr.microsoft.com/dotnet/sdk:10.0 carries PowerShell 7.6 (.NET 10 +# LTS). Microsoft publishes these for development and testing only; build and maintain your own +# image for production pipelines. +test:windows: + stage: test + image: mcr.microsoft.com/dotnet/sdk:10.0-windowsservercore-ltsc2022 + <<: *powershell_template + script: + - pwsh -NoProfile -File ./Tests/Invoke-Tests.ps1 -TestType Unit -Environment CI -CodeCoverage + artifacts: + when: always + reports: + junit: Tests/Results/TestResults*.xml + coverage_report: + coverage_format: cobertura + path: Tests/Results/Coverage*.xml + paths: + - Tests/Results/ + expire_in: 1 week + coverage: '/Code Coverage: (\d+\.?\d*)%/' + +test:linux: + stage: test + image: mcr.microsoft.com/dotnet/sdk:10.0 + <<: *powershell_template + script: + # This job measures no coverage, so it can run in parallel. + - pwsh -NoProfile -File ./Tests/Invoke-Tests.ps1 -TestType Unit -Tag CrossPlatform -Environment CI -Parallel + artifacts: + when: always + reports: + junit: Tests/Results/TestResults*.xml + +security: + stage: security + image: mcr.microsoft.com/dotnet/sdk:10.0-windowsservercore-ltsc2022 + <<: *powershell_template + script: + - pwsh -NoProfile -File ./Tests/Invoke-Tests.ps1 -TestType Security -Environment CI + allow_failure: false + +deploy: + stage: deploy + image: mcr.microsoft.com/dotnet/sdk:10.0-windowsservercore-ltsc2022 + script: + - pwsh -NoProfile -Command "Write-Host 'Deploying to PowerShell Gallery...'" + only: + - main + when: manual +``` + +The `coverage:` pattern reads the figure from the line `Code Coverage:
%` that the runner +prints in its summary, and GitLab shows it on the merge request. It reads the runner's own line, +not the output of Pester. + +Check which PowerShell version an image contains before you rely on it; the `#requires` line of the +script fails the job when the version is too old. + +As in Azure Pipelines, the deployment jobs of Jenkins and GitLab run from the main branch after the +tests. CIC-08 is narrower for publishing a module: only from a tagged release. + +## Shared Settings + +Set the same environment variables in every pipeline: + +```powershell +# Common environment variables for CI +$env:POWERSHELL_TELEMETRY_OPTOUT = '1' +$env:PESTER_CI_MODE = 'true' +$env:CODE_COVERAGE_ENABLED = 'true' +$env:TEST_RESULTS_PATH = './Tests/Results' +``` + +`POWERSHELL_TELEMETRY_OPTOUT` stops PowerShell from sending telemetry from the build agent. Pester +does not read the other three; they are shared names for the scripts of the pipeline. Pin the Pester +version in one variable, `PESTER_VERSION`, as the GitHub Actions workflow does. + +Collect the reports with the same patterns in every pipeline: + +```powershell +# Standard artifact patterns +$artifacts = @{ + TestResults = './Tests/Results/TestResults*.xml' + CodeCoverage = './Tests/Results/Coverage*.xml' + TestMetrics = './Tests/Results/TestMetrics*.json' + TestReports = './Tests/Results/TestReport*.html' +} +``` ## Report Formats | CI system | Test result format | Coverage format | | --- | --- | --- | -| GitHub Actions | Any. The reports are uploaded as an artifact. | Any | +| GitHub Actions | `NUnitXml`, shown with `dorny/test-reporter`. Alternative: any format, uploaded as an artifact. | `JaCoCo` | | Azure Pipelines | `NUnitXml`, published with `PublishTestResults@2` | `JaCoCo` or `Cobertura`, published with `PublishCodeCoverageResults@2` | | GitLab CI | `JUnitXml` | `Cobertura` | -| Jenkins | `JUnitXml`, published with the `junit` step | `JaCoCo` or `Cobertura`, depending on the plugin | +| Jenkins | `JUnitXml`, published with the `junit` step of the JUnit plugin | `JaCoCo` or `Cobertura`, depending on the plugin | +| Codecov | Any | `JaCoCo` or `Cobertura` | Pester writes one test result format for each run. To write a second one, convert the result object with `Export-NUnitReport` or `Export-JUnitReport`; see [Pester Configuration Guide](./pester-configuration.md#test-results). +The `CoverageGutters` format was removed. Every coverage report is now relative to the repository +root, so the plain `JaCoCo` format works with the Coverage Gutters extension. + Paths inside a coverage report are relative to `CodeCoverage.ReportRoot`, which defaults to -`Run.RepoRoot`. Set `ReportRoot` when the CI system expects paths relative to another folder. +`Run.RepoRoot`. Set `ReportRoot` when the CI system checks out to a subfolder or expects paths +relative to another folder. ## Failing Correctly @@ -393,35 +1061,71 @@ if ($result.FailedCount -gt 0) { } ``` -Do not turn a failure into a warning to keep a build green. A pull request pipeline fails when -Pester reports a problem (CIC-01). Tests that need resources a pull request build does not have, -such as integration tests against a live service, belong in a separate workflow that also fails when -they fail. +The analyzer, the unit tests, the security tests, and the coverage figure are the gate: a pull +request pipeline fails when one of them reports a problem (CIC-01). Do not turn a failure of the +gate into a warning to keep a build green. + +Integration tests and performance tests report instead: their steps write a warning when a test +fails, and the build goes on. Those tests depend on external resources and on the load of the +runner. The alternative is to let them fail the build as well: call the runner without `-PassThru`, +or put tests that need resources a pull request build does not have in a separate workflow that +fails when they fail. -`Invoke-Pester` sets `$LASTEXITCODE` to the number of failures. Do not reset it before the script -has thrown for the failure it carried (CIC-05). +`Invoke-Pester` sets `$LASTEXITCODE` to the number of failures, and the `pwsh` shell of GitHub +Actions fails a step whose last exit code is not zero. Do not reset it before the script has acted +on the failure it carried (CIC-05): the gate throws, and the two steps that only warn reset it +after they have written the warning. ## Parallel Runs and Coverage in CI -`Run.Parallel` is experimental, so the gate script does not use it. When a project chooses to use -it, keep the coverage gate in a sequential job: a parallel run measures coverage with the slower -breakpoint method. Put the parallel run in a second job without coverage. Details are in +Pester 6 has a parallel runner, `Run.Parallel`. `Run.Container` is for passing data to test files, +not for parallelism: measured on 6.2.0, three containers that each took two seconds ran one after +another in the calling runspace in nine seconds, and in three seconds with `Run.Parallel` set. + +```powershell +$configuration = New-PesterConfiguration +$configuration.Run.Path = $testRoot +$configuration.Run.Parallel = $true +$configuration.Run.ParallelThrottleLimit = 4 +``` + +Containers with data still work, and they run in parallel too. The data of each file reaches its +worker: + +```powershell +$configuration.Run.Container = @( + New-PesterContainer -Path $testRoot -Data @{ Environment = 'CI' } +) +$configuration.Run.Parallel = $true +``` + +Splitting into two jobs is the default: a parallel job without coverage for feedback, and a +sequential job with the coverage gate. A parallel run measures coverage with breakpoints, not with +the profiler, so keep the coverage gate in the sequential job; but measure before assuming that +either is faster. + +`Run.Parallel` is experimental. The alternative, for a project that does not turn it on, is the one +sequential job of the gate script. Details are in [Test Execution Guide](./test-execution.md#parallel-test-execution). ## Settings Files in CI -When the gate script loads Pester settings from a `.psd1` file, a misspelled key only produces a +When the script loads Pester settings from a `.psd1` file, a misspelled key only produces a warning. Fail the build on it with `GetUnknownKeys()`; see -[Pester Configuration Guide](./pester-configuration.md#checking-what-was-loaded). +[Pester Configuration Guide](./pester-configuration.md#checking-what-was-loaded). The runner does +this through `Test-PesterConfiguration`. ## Checklist - Tool versions are pinned: Pester 6.2.0 and PSScriptAnalyzer 1.25.0. -- The pipeline calls one gate script, and the script tests for no tests, for `Result`, and for +- The pipeline calls one script, and the script tests for no tests, for `Result`, and for coverage. +- A discovery-only job runs first, and the other jobs depend on it. +- Coverage is measured in a sequential job; a parallel job measures none. - Every action is referenced by a release tag or a full commit ID (CIC-09). - The test folder is named in one place. - Every script sets `$ErrorActionPreference = 'Stop'` and fails by throwing. - The workflow has an explicit `permissions` block with the least access it needs (CIC-07). - No secret is written in the workflow or printed to the log (CIC-06). - No job targets a PowerShell version that is past its end-of-support date. +- Reports are uploaded even when the run fails. diff --git a/.github/instructions/pester-supporting-docs/custom-assertions.md b/.github/instructions/pester-supporting-docs/custom-assertions.md index eafb377..fadd088 100644 --- a/.github/instructions/pester-supporting-docs/custom-assertions.md +++ b/.github/instructions/pester-supporting-docs/custom-assertions.md @@ -244,7 +244,8 @@ carries no hint. Test the value first and call `Fail()`. `Should` is not an approved verb. A module that exports a function named `Should-*` makes `Import-Module` warn about unapproved verbs, on PowerShell 7 and Windows PowerShell 5.1, and listing -the function in `FunctionsToExport` does not prevent the warning. +the function in `FunctionsToExport` does not prevent the warning. `-DisableNameChecking` only moves +the problem onto the module's users: the warning is back for everyone who imports without it. Name the function with the approved verb `Assert` and export a `Should-*` alias. Aliases are not checked: @@ -264,8 +265,9 @@ The root module holds the functions and the `Set-Alias` lines shown above. Peste `$PSCmdlet` passed as `-Caller`, not from the name of the command, so the assertion behaves the same when it is called through the alias. -Use the same names when a test file defines the function itself. No warning appears there, but -STY-10 asks for an approved verb in every function name. +This applies only to modules. A test file that defines or dot-sources a `Should-*` function directly +gets no warning and needs none of it. Where PSScriptAnalyzer checks the test files, its +`PSUseApprovedVerbs` rule still reports the name (STY-10). ## Where To Put Them @@ -307,7 +309,7 @@ BeforeAll { Import-Module -Name (Join-Path -Path $PSScriptRoot -ChildPath 'Assertions.psd1') -Force } -Describe 'Should-BeValidSemVer' -Tag 'Unit' { +Describe 'Should-BeValidSemVer' -Tag 'Unit', 'TestHelpers' { It 'Passes for ' -ForEach @( @{ Version = '1.2.3' } @{ Version = '0.0.1' } @@ -349,5 +351,9 @@ ends the test with the assertion's own message: ```powershell Should-Invoke -CommandName Publish-Module -ModuleName ModuleName -Times 1 -Exactly -ParameterFilter { $RequiredVersion | Should-BeValidSemVer + $true } ``` + +The trailing `$true` keeps the result of the filter explicit when the assertion is not the last +statement. diff --git a/.github/instructions/pester-supporting-docs/integration-test-template.md b/.github/instructions/pester-supporting-docs/integration-test-template.md index 81d263a..6c94d4e 100644 --- a/.github/instructions/pester-supporting-docs/integration-test-template.md +++ b/.github/instructions/pester-supporting-docs/integration-test-template.md @@ -1,9 +1,14 @@ # Integration Test Template -An integration test file: the module runs against the real file system and a real service, with -nothing mocked. The rules it follows are in the [Pester rules](../powershell-pester.instructions.md). -The file below passes on Pester 6.2.0 under PowerShell 7.6 and Windows PowerShell 5.1, with the -service configured and without it. +Use this template for integration tests that validate how components work together. TST-43 requires +them for code that has external dependencies. In an integration test file the module runs against +the real file system and a real service, with nothing mocked. The rules it follows are in the +[Pester rules](../powershell-pester.instructions.md). The files below pass on Pester 6.2.0 under +PowerShell 7.6 and Windows PowerShell 5.1: with the service running, with the service configured +but not answering, and without it. + +Do not use emoji in generated code, documentation, or test output. Use plain text and standard +ASCII characters only (STY-06, TST-07). ## The Code Under Test @@ -14,6 +19,7 @@ The test file is in the `Tests` folder directly under the module root. without a byte order mark. The file holds a `savedAt` time and a `things` array. The function supports `-WhatIf`. - `Get-Thing -Name -ServiceUri ` reads things from the service at that address. +- `Read-ThingFile -Root -ChildPath ` reads a file below a root folder. The address of a test instance of the service comes from the environment variable `THING_SERVICE_URI`. @@ -36,7 +42,7 @@ BeforeAll { } } -Describe 'Export-Thing with the real file system' -Tag 'Integration' { +Describe 'Export-Thing with the real file system' -Tag 'Integration', 'SystemIntegration' { BeforeAll { $things = @( [PSCustomObject]@{ PSTypeName = 'ModuleName.Thing'; Name = 'alpha'; State = 'ready' } @@ -44,7 +50,7 @@ Describe 'Export-Thing with the real file system' -Tag 'Integration' { ) } - It 'Writes a file that reads back as the same things' { + It 'Should write a file that reads back as the same things' { $exportPath = Join-Path -Path $TestDrive -ChildPath 'things.json' $things | Export-Thing -LiteralPath $exportPath @@ -59,7 +65,7 @@ Describe 'Export-Thing with the real file system' -Tag 'Integration' { $document | Should-BeEquivalent $expected -ExcludePath 'savedAt' } - It 'Writes UTF-8 without a byte order mark' { + It 'Should write UTF-8 without a byte order mark' { $exportPath = Join-Path -Path $TestDrive -ChildPath 'encoding.json' $things | Export-Thing -LiteralPath $exportPath @@ -70,7 +76,7 @@ Describe 'Export-Thing with the real file system' -Tag 'Integration' { $bytes[0] | Should-Be 123 } - It 'Writes nothing when -WhatIf is passed' { + It 'Should write nothing when -WhatIf is passed' { $exportPath = Join-Path -Path $TestDrive -ChildPath 'whatif.json' $things | Export-Thing -LiteralPath $exportPath -WhatIf @@ -78,7 +84,7 @@ Describe 'Export-Thing with the real file system' -Tag 'Integration' { Test-Path -LiteralPath $exportPath | Should-BeFalse } - It 'Throws when the folder does not exist' { + It 'Should throw when the folder does not exist' { $missingFolder = Join-Path -Path $TestDrive -ChildPath 'missing' $exportPath = Join-Path -Path $missingFolder -ChildPath 'things.json' @@ -87,14 +93,50 @@ Describe 'Export-Thing with the real file system' -Tag 'Integration' { $failure.Exception.InnerException | Should-HaveType ([System.IO.DirectoryNotFoundException]) } + + It 'Should handle path separators correctly' { + $reportFolder = Join-Path -Path $TestDrive -ChildPath 'reports' + $null = New-Item -Path $reportFolder -ItemType Directory + $reportPath = Join-Path -Path $reportFolder -ChildPath 'report.txt' + Set-Content -LiteralPath $reportPath -Value 'report' -Encoding ascii + $childPath = Join-Path -Path 'reports' -ChildPath 'report.txt' + + Read-ThingFile -Root $TestDrive -ChildPath $childPath | Should-BeString 'report' -TrimWhitespace + + # $IsWindows does not exist on Windows PowerShell 5.1, which always runs on Windows. + $onWindows = $PSVersionTable.PSEdition -eq 'Desktop' -or $IsWindows + if ($onWindows) { + $childPath | Should-MatchString '\\' + } else { + $childPath | Should-MatchString '/' + } + } } -Describe 'Get-Thing with the real service' -Tag 'Integration' -Skip:$serviceIsMissing { +Describe 'Get-Thing with the real service' -Tag 'Integration', 'SystemIntegration' -Skip:$serviceIsMissing { BeforeAll { - $serviceUri = $env:THING_SERVICE_URI + $serviceUri = [uri]$env:THING_SERVICE_URI + + # Verify once that the service can be reached. + $serviceError = $null + $client = [System.Net.Sockets.TcpClient]::new() + try { + $client.Connect($serviceUri.Host, $serviceUri.Port) + } catch [System.Management.Automation.MethodInvocationException] { + # The tests of this block are skipped with this text as the reason. + $serviceError = $_.Exception.InnerException.Message + } finally { + $client.Dispose() + } } - It 'Reads a thing that exists' { + BeforeEach { + if ($null -ne $serviceError) { + Set-ItResult -Skipped -Because "the external service is not available: $serviceError" + } + } + + It 'Should read a thing that exists' { $result = @(Get-Thing -Name 'alpha' -ServiceUri $serviceUri) $result | Should-BeCollection -Count 1 @@ -102,7 +144,7 @@ Describe 'Get-Thing with the real service' -Tag 'Integration' -Skip:$serviceIsMi $result[0].Name | Should-Be 'alpha' } - It 'Reports a thing that does not exist as one non-terminating error' { + It 'Should report a thing that does not exist as one non-terminating error' { $output = Get-Thing -Name 'no-such-thing' -ServiceUri $serviceUri 2>&1 $errors = @($output | Where-Object { $_ -is [System.Management.Automation.ErrorRecord] }) @@ -110,6 +152,362 @@ Describe 'Get-Thing with the real service' -Tag 'Integration' -Skip:$serviceIsMi $errors | Should-BeCollection -Count 1 $errors[0].FullyQualifiedErrorId | Should-Be 'ThingReadFailed,Get-Thing' } + + Context 'End-to-end workflow' -Tag 'EndToEnd' { + It 'Should complete the full workflow' { + $exportPath = Join-Path -Path $TestDrive -ChildPath 'workflow.json' + + Get-Thing -Name 'alpha' -ServiceUri $serviceUri -ErrorAction Stop | + Export-Thing -LiteralPath $exportPath + + $document = Get-Content -LiteralPath $exportPath -Raw | ConvertFrom-Json + $document.things | Should-BeCollection -Count 1 + $document.things[0].name | Should-Be 'alpha' + } + + It 'Should handle partial failures gracefully' { + $exportPath = Join-Path -Path $TestDrive -ChildPath 'partial.json' + + $output = Get-Thing -Name 'alpha', 'no-such-thing' -ServiceUri $serviceUri -ErrorAction Continue 2>&1 + $errors = @($output | Where-Object { $_ -is [System.Management.Automation.ErrorRecord] }) + $things = @($output | Where-Object { $_ -isnot [System.Management.Automation.ErrorRecord] }) + $things | Export-Thing -LiteralPath $exportPath + + # The workflow completed, and the failure was reported for the one thing. + $errors | Should-BeCollection -Count 1 + $document = Get-Content -LiteralPath $exportPath -Raw | ConvertFrom-Json + $document.things | Should-BeCollection -Count 1 + } + + It 'Should maintain data consistency across components' { + $exportPath = Join-Path -Path $TestDrive -ChildPath 'consistent.json' + + $thing = Get-Thing -Name 'alpha' -ServiceUri $serviceUri -ErrorAction Stop + $thing | Export-Thing -LiteralPath $exportPath + + $document = Get-Content -LiteralPath $exportPath -Raw | ConvertFrom-Json + $expected = [PSCustomObject]@{ name = $thing.Name; state = $thing.State } + $document.things[0] | Should-BeEquivalent $expected + } + } +} +``` + +## Tests Against Stand-In Functions + +The example module has no retry logic and no data store. The tests in this section therefore run +against a small stand-in module that the template defines itself. In a project, the functions under +test are the project's own. Comment-based help is omitted here to keep the example short. + +```powershell +# Tests/TestHelpers/IntegrationSample.psm1 +Set-StrictMode -Version Latest + + +function Invoke-WithRetry { + [CmdletBinding()] + [OutputType('IntegrationSample.RetryResult')] + param( + [Parameter(Mandatory)] + [scriptblock]$ScriptBlock, + + [ValidateRange(1, 10)] + [int]$MaxRetries = 3 + ) + + $attemptCount = 0 + $succeeded = $false + $value = $null + while (-not $succeeded -and $attemptCount -lt $MaxRetries) { + $attemptCount++ + try { + $value = & $ScriptBlock + $succeeded = $true + } catch [System.TimeoutException] { + # A time-out is the transient failure that this function retries. + Write-Verbose "Attempt $attemptCount timed out." + } + } + + [PSCustomObject]@{ + PSTypeName = 'IntegrationSample.RetryResult' + Success = $succeeded + AttemptCount = $attemptCount + Value = $value + } +} + + +function Invoke-WithCircuitBreaker { + [CmdletBinding()] + [OutputType('IntegrationSample.CircuitResult')] + param( + [Parameter(Mandatory)] + [scriptblock]$ScriptBlock, + + [ValidateRange(1, 10)] + [int]$FailureThreshold = 3 + ) + + $stopwatch = [System.Diagnostics.Stopwatch]::StartNew() + $failureCount = 0 + $succeeded = $false + while (-not $succeeded -and $failureCount -lt $FailureThreshold) { + try { + $null = & $ScriptBlock + $succeeded = $true + } catch [System.Net.WebException] { + # A failed request counts towards the threshold that opens the circuit. + $failureCount++ + } + } + $stopwatch.Stop() + + [PSCustomObject]@{ + PSTypeName = 'IntegrationSample.CircuitResult' + CircuitOpen = -not $succeeded + FailureCount = $failureCount + ExecutionTime = $stopwatch.Elapsed + } +} + + +function Save-Record { + [CmdletBinding(SupportsShouldProcess)] + [OutputType('IntegrationSample.SaveResult')] + param( + [Parameter(Mandatory)] + [string]$StorePath, + + [Parameter(Mandatory)] + [hashtable]$Data, + + [string]$Id = [guid]::NewGuid().ToString('N'), + + [string]$ParentId = '' + ) + + $recordPath = Join-Path -Path $StorePath -ChildPath "$Id.json" + if (-not $PSCmdlet.ShouldProcess($recordPath, 'Save record')) { + return + } + + $record = [ordered]@{ id = $Id; parentId = $ParentId; data = $Data } + $json = ConvertTo-Json -InputObject $record -Depth 5 + + # One writer at a time, in this process and in others, so that a record is never half written. + $mutex = [System.Threading.Mutex]::new($false, 'IntegrationSampleStore') + $null = $mutex.WaitOne() + try { + [System.IO.File]::WriteAllText($recordPath, $json, [System.Text.UTF8Encoding]::new($false)) + } catch { + $PSCmdlet.ThrowTerminatingError($_) + } finally { + $mutex.ReleaseMutex() + $mutex.Dispose() + } + + [PSCustomObject]@{ + PSTypeName = 'IntegrationSample.SaveResult' + Success = $true + RecordId = $Id + } +} + + +function Get-Record { + [CmdletBinding(DefaultParameterSetName = 'ById')] + [OutputType('IntegrationSample.Record')] + param( + [Parameter(Mandatory)] + [string]$StorePath, + + [Parameter(Mandatory, ParameterSetName = 'ById')] + [string]$Id, + + [Parameter(Mandatory, ParameterSetName = 'ByParent')] + [string]$ParentId + ) + + $byId = $PSCmdlet.ParameterSetName -eq 'ById' + $filter = if ($byId) { "$Id.json" } else { '*.json' } + foreach ($file in Get-ChildItem -LiteralPath $StorePath -Filter $filter -File) { + $json = Get-Content -LiteralPath $file.FullName -Raw + $record = ConvertFrom-Json -InputObject $json + if ($byId -or $record.parentId -eq $ParentId) { + [PSCustomObject]@{ + PSTypeName = 'IntegrationSample.Record' + Id = $record.id + ParentId = $record.parentId + Data = $record.data + } + } + } +} + + +function Remove-Record { + [CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'High')] + param( + [Parameter(Mandatory)] + [string]$StorePath, + + [Parameter(Mandatory)] + [string]$Id, + + [switch]$Cascade + ) + + $recordIds = @($Id) + if ($Cascade) { + $recordIds = @( + foreach ($child in Get-Record -StorePath $StorePath -ParentId $Id) { + $child.Id + } + $Id + ) + } + + foreach ($recordId in $recordIds) { + $recordPath = Join-Path -Path $StorePath -ChildPath "$recordId.json" + if ($PSCmdlet.ShouldProcess($recordPath, 'Remove record')) { + try { + Remove-Item -LiteralPath $recordPath -ErrorAction Stop + } catch { + $PSCmdlet.ThrowTerminatingError($_) + } + } + } +} +``` + +The test file for these functions: + +```powershell +#Requires -Modules @{ ModuleName = 'Pester'; ModuleVersion = '6.2.0' } + +#pester:no-parallel + +BeforeAll { + $helperPath = Join-Path -Path $PSScriptRoot -ChildPath 'TestHelpers' + $samplePath = Join-Path -Path $helperPath -ChildPath 'IntegrationSample.psm1' + if (-not (Get-Module -Name IntegrationSample)) { + Import-Module -Name $samplePath + } +} + +Describe 'Calls to an unreliable service' -Tag 'Integration', 'SystemIntegration' { + It 'Should retry on transient failures' { + # The script block fails twice and then answers. + $calls = @{ Count = 0 } + $operation = { + $calls.Count++ + if ($calls.Count -lt 3) { + throw [System.TimeoutException]::new('The service did not answer.') + } + 'answer' + } + + $retryResult = Invoke-WithRetry -ScriptBlock $operation -MaxRetries 3 + + $retryResult.Success | Should-BeTrue + $retryResult.Value | Should-Be 'answer' + $retryResult.AttemptCount | Should-BeGreaterThan 1 + $retryResult.AttemptCount | Should-BeLessThanOrEqual 3 -Because 'MaxRetries caps the attempts at 3' + } + + It 'Should circuit break on persistent failures' { + $calls = @{ Count = 0 } + $operation = { + $calls.Count++ + throw [System.Net.WebException]::new('The service cannot be reached.') + } + + $circuitResult = Invoke-WithCircuitBreaker -ScriptBlock $operation -FailureThreshold 3 + + $circuitResult.CircuitOpen | Should-BeTrue + $calls.Count | Should-Be 3 -Because 'an open circuit stops the calls' + $circuitResult.ExecutionTime | Should-BeLessThan ([timespan]::FromSeconds(10)) + } +} + +Describe 'Data persistence' -Tag 'Integration', 'SystemIntegration' { + BeforeAll { + $storePath = Join-Path -Path $TestDrive -ChildPath 'store' + $null = New-Item -Path $storePath -ItemType Directory + } + + It 'Should persist data correctly' { + $testData = @{ + Name = 'IntegrationTest' + Value = 'TestValue' + Timestamp = [datetime]::new(2026, 3, 31, 12, 0, 0) + ComplexData = @{ + NestedProperty = 'NestedValue' + ArrayProperty = @(1, 2, 3) + } + } + + $saveResult = Save-Record -StorePath $storePath -Data $testData + $saveResult.Success | Should-BeTrue + $saveResult.RecordId | Should-MatchString '^[0-9a-f]{32}$' + + $record = Get-Record -StorePath $storePath -Id $saveResult.RecordId + + # One deep comparison reports every property that the round trip changed. Timestamp is + # excluded because storage does not return it unchanged on every edition. + $expected = [PSCustomObject]@{ + Name = 'IntegrationTest' + Value = 'TestValue' + Timestamp = $testData.Timestamp + ComplexData = [PSCustomObject]@{ + NestedProperty = 'NestedValue' + ArrayProperty = @(1, 2, 3) + } + } + $record.Data | Should-BeEquivalent $expected -ExcludePath 'Timestamp' + } + + It 'Should handle concurrent data access' { + $recordId = 'shared-record' + $jobScript = { + param($SamplePath, $StorePath, $RecordId, $WorkerId) + + Import-Module -Name $SamplePath + Save-Record -StorePath $StorePath -Id $RecordId -Data @{ Value = "Worker$WorkerId" } + } + + $jobs = foreach ($workerId in 1..5) { + Start-Job -ScriptBlock $jobScript -ArgumentList $samplePath, $storePath, $recordId, $workerId + } + $results = @($jobs | Wait-Job | Receive-Job) + $jobs | Remove-Job + + # Every writer finished, and the record is the complete record of one of them. + $results | Should-BeCollection -Count 5 + $record = Get-Record -StorePath $storePath -Id $recordId + $record.Data.Value | Should-MatchString '^Worker[1-5]$' + } + + It 'Should maintain referential integrity' { + $parentId = 'parent-1' + $null = Save-Record -StorePath $storePath -Id $parentId -Data @{ Name = 'TestParent' } + foreach ($number in 1..3) { + $child = @{ Name = "TestChild$number" } + (Save-Record -StorePath $storePath -ParentId $parentId -Data $child).Success | Should-BeTrue + } + + # The relationships are stored. + $children = @(Get-Record -StorePath $storePath -ParentId $parentId) + $children | Should-BeCollection -Count 3 + $children | Should-All { $_.ParentId -eq $parentId } + + # Removing the parent with -Cascade removes the children. + Remove-Record -StorePath $storePath -Id $parentId -Cascade -Confirm:$false + + @(Get-Record -StorePath $storePath -ParentId $parentId).Count | Should-Be 0 + Get-Record -StorePath $storePath -Id $parentId | Should-BeNull + } } ``` @@ -122,18 +520,56 @@ removes it afterwards, so the tests need no cleanup code and cannot leave files else that a test changes, such as an environment variable or a record in the service, must be restored in `AfterAll` or `AfterEach` (TST-31). +A resource that cannot be in `$TestDrive`, such as a working folder that the code under test looks +for in a fixed place, is created in `BeforeAll` and removed in `AfterAll`: + +```powershell +BeforeAll { + # Create the test resources. + $workFolder = Join-Path -Path ([System.IO.Path]::GetTempPath()) -ChildPath 'IntegrationTests' + if (-not (Test-Path -LiteralPath $workFolder)) { + $null = New-Item -Path $workFolder -ItemType Directory + } +} + +AfterAll { + # Clean up the test resources. + if (Test-Path -LiteralPath $workFolder) { + Remove-Item -LiteralPath $workFolder -Recurse -Force + } +} +``` + ### A Missing Environment Is Skipped At Discovery -`-Skip:` takes a value that is known during discovery, so it is computed in `BeforeDiscovery`. The -tests of a skipped block are reported as `Skipped`, and the block's `BeforeAll` does not run. +Take the configuration of the test environment from environment variables. `-Skip:` takes a value +that is known during discovery, so it is computed in `BeforeDiscovery`. The tests of a skipped +block are reported as `Skipped`, and the block's `BeforeAll` does not run. + +### An Unreachable Service Is Skipped At Run Time + +When the address is set, `BeforeAll` verifies once that the service can be reached, and +`BeforeEach` calls `Set-ItResult -Skipped -Because ''` for every test of the block when it +cannot. Run with an address where nothing listens, the five service tests were reported as +`Skipped` with the connection error as the reason, and the run passed. + +A project that wants an outage of a configured service to fail the build leaves out the check. +The tests then fail when the service does not answer, which a skip would hide. + +A test can mark itself the same way from the `It` block, with `-Skipped` or `-Inconclusive`. +`Set-ItResult -Pending` was removed in Pester 6. + +### Real Services In A Controlled Environment + +Use real services in a controlled test environment, and mock only when a real service is not +available. The retry and circuit-breaker tests show the second case: no real service fails on +request, so a script block that fails a set number of times stands in for it. -Skip only for an environment that is not configured. When the address is set and the service does -not answer, the tests must fail; a skip there would hide the outage. A test can still mark itself at -run time with `Set-ItResult -Skipped -Because ''` or `Set-ItResult -Inconclusive`, from the -`It` block or from `BeforeEach`. `Set-ItResult -Pending` was removed in Pester 6. +### Tags -Tag the `Describe` blocks `Integration` (TST-04), so that a run can leave them out with -`Filter.ExcludeTag`. +Tag the `Describe` blocks `Integration` and with the category, `EndToEnd` or `SystemIntegration` +(TST-04), so that a run can leave them out with `Filter.ExcludeTag`. In the template the workflow +tests need the service block's setup, so their category tag is on the `Context`. ### Test Cases From The Environment @@ -145,8 +581,10 @@ legitimate. ### Whole Objects Are Compared With Should-BeEquivalent One `Should-BeEquivalent` reports every property that differs after a round trip. `-ExcludePath` -leaves out a value that changes on every run, here `savedAt`. `Should-BeEquivalent` does not compare -the order of a collection; add index assertions after a count when order is part of the contract. +leaves out a value that changes on every run, here `savedAt`, or one that storage does not return +unchanged, here `Timestamp`: on Windows PowerShell 5.1 the JSON round trip returned that time in +UTC, and the comparison failed without the exclusion. `Should-BeEquivalent` does not compare the +order of a collection; add index assertions after a count when order is part of the contract. ### A .NET Failure Arrives Wrapped @@ -155,12 +593,36 @@ the order of a collection; add index assertions after a count when order is part which is a `MethodInvocationException`. The test asserts that type and then reads the original exception from the error record that `Should-Throw` returns. +### Workflows, Partial Failures, And Consistency + +The workflow tests run the functions together, as a caller would: read from the service, then write +the file. One test completes the workflow, one injects a failure for part of the input and asserts +that the rest still arrives and the failure is reported, and one compares what the first component +returned with what the last one stored. + +### Retry And Circuit Breaker + +The retry test asserts that the call succeeded, that more than one attempt was made, and that the +number of attempts stayed within `MaxRetries`. The circuit-breaker test asserts that the circuit is +open, that the calls stopped at the threshold, and that the whole call returned quickly. The time +limit is a `[timespan]` compared with a `[timespan]`. + +### Persistence + +The persistence tests cover the three things that a store must get right: a record reads back as +it was saved, concurrent writers leave a whole record, and child records stay with their parent +and are removed with it. The concurrency test starts five jobs. A job is a new process, so the +script block receives every path as an argument and imports the module itself. + ### The File Does Not Run In Parallel -The comment `#pester:no-parallel` keeps the file out of the worker runspaces when a run sets -`Run.Parallel`; such files run one after another in the main session. It has no effect otherwise. -`Run.Parallel` is experimental and off by default. Mark a file this way when it uses a resource that -two files cannot share, such as a fixed port or a record in a test service. +Integration tests hold shared external resources: a database, a fixed port, a live endpoint. Two +files that touch the same resource at the same time produce failures that are hard to diagnose. +Put the comment `#pester:no-parallel` at the top of every integration test file. It keeps the file +out of the worker runspaces when a run sets `Run.Parallel`; such files run one after another in the +main session. It has no effect otherwise. `Run.Parallel` is experimental and off by default. + +A file whose tests touch nothing but their own `$TestDrive` can leave the comment out. ### Both Editions @@ -170,8 +632,47 @@ The file runs unchanged on Windows PowerShell 5.1 because it avoids what differs calls. - `$IsWindows` does not exist on 5.1, and reading it under strict mode is an error. Test `$PSVersionTable.PSEdition -eq 'Desktop' -or $IsWindows`, which reads it only on PowerShell 7. +- The path-separator test builds its path with `Join-Path` and asserts the separator of the + platform: a backslash on Windows, a slash elsewhere. - The byte order mark is asserted from the bytes of the file, not from the encoding name, because `-Encoding utf8` means different things on the two editions (STY-33). To limit a test to one edition, pass a discovery-time condition such as `-Skip:($PSVersionTable.PSEdition -ne 'Core')`. + +## Integration Test Guidelines + +### Test Environment Setup + +- Use environment variables for configuration. +- Create isolated test resources. +- Verify that external services are available. +- Clean up resources after the tests. + +### End-to-End Testing + +- Test complete business workflows. +- Validate the interactions between components. +- Verify data flow and consistency. +- Test error propagation and recovery. + +### External Dependencies + +- Use real services in a controlled environment. +- Test connectivity and error handling. +- Validate retry and circuit-breaker patterns. +- Mock only when real services are not available. + +### Data Persistence Testing + +- Test create, read, update, and delete operations. +- Verify data integrity and consistency. +- Test concurrent access. +- Validate referential integrity. + +### Cross-Platform Considerations + +- Test on every PowerShell version that the project supports. +- Verify the differences in path handling. +- Test platform-specific features. +- Use conditional test execution. diff --git a/.github/instructions/pester-supporting-docs/mocking-patterns.md b/.github/instructions/pester-supporting-docs/mocking-patterns.md index 59e7219..a9bf4a8 100644 --- a/.github/instructions/pester-supporting-docs/mocking-patterns.md +++ b/.github/instructions/pester-supporting-docs/mocking-patterns.md @@ -1,6 +1,6 @@ # Mocking Patterns Guide -Patterns for `Mock`, `Should-Invoke`, and `InModuleScope`. The rules are TST-20 to TST-22 in the +Patterns for `Mock`, `Should-Invoke`, and `InModuleScope`. The rules are TST-20 to TST-24 in the [Pester rules](../powershell-pester.instructions.md); this guide holds the detail behind them. Every statement about behaviour was checked by running it on Pester 6.2.0. @@ -59,6 +59,9 @@ Performed invocations: A filter may name a parameter by one of its aliases. A call to `Get-Content -LiteralPath` matches `-ParameterFilter { $PSPath -like '*settings*' }`. +Mocking also falls back gracefully when a command cannot produce its dynamic parameters. A function +whose `dynamicparam` block throws is still replaced by its mock. + ## Where A Mock Applies A mock replaces a command for one caller (TST-21): @@ -88,10 +91,41 @@ It 'Appends the path of the thing to the service address' { } ``` +A mock declared inside `InModuleScope` applies inside the module and needs no `-ModuleName`. It is +the other way to replace a private function that a public one calls: + +```powershell +It 'Returns nothing when the status service cannot be reached' { + InModuleScope -ModuleName ModuleName -ScriptBlock { + Mock -CommandName Get-ServiceStatus -MockWith { throw 'service unreachable' } + + $result = 'Billing', 'Identity' | Get-Data -ErrorAction SilentlyContinue + + $result | Should-BeFalsy + Should-Invoke -CommandName Get-ServiceStatus -Times 2 -Exactly + } +} +``` + +`InModuleScope` is also how a test reaches a class that the module defines, for example in its +`Classes` folder. A test file that imports the module does not see the type: there `[ThingCache]` +fails with `Unable to find type`. + +```powershell +It 'Keeps the state of a thing that was added to the cache' { + InModuleScope -ModuleName ModuleName -ScriptBlock { + $cache = [ThingCache]::new() + $cache.Add('alpha', 'ready') + + $cache.GetState('alpha') | Should-Be 'ready' + } +} +``` + ### How Long A Mock Lasts -A mock lasts until the block that declared it ends, and Pester then restores the command. No cleanup -code is needed. +A mock lasts until the block that declared it ends, and Pester then restores the command. The mock +itself needs no cleanup code; state that mocks share does, as "Mocks That Share State" shows. | Declared in | Applies to | | --- | --- | @@ -107,11 +141,50 @@ one test does not see the calls of another. `-Scope Describe` widens the count t A mock never reaches another test file. Pester 6 discovers and runs one file, then the next, and removes a file's mocks when the file ends. +### One Context, One Set Of Mocks + +Keep mocks isolated between tests: each `Context` declares the mocks that its tests call in its own +`BeforeEach` (TST-24): + +```powershell +Describe 'Get-ServiceHealth' -Tag 'Unit', 'Public' { + Context 'When the service answers' { + BeforeEach { + Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { + [PSCustomObject]@{ status = 'healthy' } + } + } + + It 'Returns the status that the service reported' { + Get-ServiceHealth | Should-Be 'healthy' + } + } + + Context 'When the service is unavailable' { + BeforeEach { + Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { + throw 'Service unavailable' + } + } + + It 'Fails with the error of the service' { + { Get-ServiceHealth } | Should-Throw -ExceptionMessage '*Service unavailable*' + } + } +} +``` + +From 6.2 a block may also hold several `BeforeEach` blocks, which run in the order they are written, +so a second group of mocks can be a block of its own. Separate `Context` blocks remain the better +split when the two groups serve different tests. Pester 6.0 and 6.1 stop with +`BeforeEach is already defined in this block` at the second one. + ### Mocks And Setup Blocks A `BeforeAll` in a `Context` runs before any `BeforeEach`. A mock defined in the parent's `BeforeEach` is therefore not active while that `BeforeAll` runs, and the command it calls runs for -real. Define a mock that setup code depends on in an enclosing `BeforeAll` (TST-23). +real. Define a mock that setup code depends on in an enclosing `BeforeAll` (TST-23), and keep +`BeforeEach` for mocks that only the tests themselves call. `Should-Invoke` counts the calls made inside the current `It`. A call made in `BeforeAll` is outside every `It`, so assert it with `-Scope Context`. Both were measured on Pester 6.2.0; the Pester rules @@ -146,6 +219,34 @@ The parameters of the mocked command are variables inside `-MockWith` and `-Para `param()` block is not needed. Return objects with the shape and types of the real command's output, so that the code under test reads them the way it reads the real thing. +One mock can also hold every case, in a `switch` on its arguments. The `default` branch throws for a +request that the test did not expect: + +```powershell +BeforeAll { + Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { + switch -Regex ($Uri.AbsolutePath) { + '^/api/things/[a-z]+$' { + if ($Method -eq 'Delete') { + return [PSCustomObject]@{ success = $true } + } + return [PSCustomObject]@{ name = $Uri.Segments[-1]; state = 'ready' } + } + '^/api/health$' { + return [PSCustomObject]@{ status = 'healthy' } + } + default { + $status = [System.Net.WebExceptionStatus]::ProtocolError + throw [System.Net.WebException]::new("Not Found: $Uri", $status) + } + } + } +} +``` + +A `param()` block that names the parameters, such as `param($Uri, $Method, $Body, $Headers)`, is +accepted at the top of such a mock body and receives the same values. + ### A Dependency That Fails Throw the exception type that the real dependency throws, and assert what the function does with it: @@ -161,6 +262,111 @@ It 'Reports a service timeout as a read error' { } ``` +A slow answer and a timeout are simulated with `Start-Sleep` in the mock. The timeout mock sleeps +past the limit that the test passes, then throws the exception of a request that timed out: + +```powershell +BeforeAll { + Mock -CommandName Invoke-WebRequest -ModuleName ModuleName -MockWith { + [PSCustomObject]@{ StatusCode = 200; Content = '{"data": "sample response"}' } + } + Mock -CommandName Invoke-WebRequest -ModuleName ModuleName -ParameterFilter { + $Uri.Host -eq 'slow.test' + } -MockWith { + Start-Sleep -Milliseconds 2000 + [PSCustomObject]@{ StatusCode = 200; Content = '{"data": "sample response"}' } + } + Mock -CommandName Invoke-WebRequest -ModuleName ModuleName -ParameterFilter { + $Uri.Host -eq 'timeout.test' + } -MockWith { + Start-Sleep -Seconds 2 + $status = [System.Net.WebExceptionStatus]::Timeout + throw [System.Net.WebException]::new('The operation has timed out', $status) + } +} + +It 'Returns the page from a slow server' { + Get-PageContent -Uri 'https://slow.test/' | Should-Be '{"data": "sample response"}' +} + +It 'Reports a timeout as a WebException' { + { Get-PageContent -Uri 'https://timeout.test/' -TimeoutSec 1 } | + Should-Throw -ExceptionType ([System.Net.WebException]) +} +``` + +The delay is written out, because the mock body cannot read `$TimeoutSec` on both editions: on +PowerShell 7.6 that name is an alias of `-ConnectionTimeoutSeconds`, and the variable does not +exist there. The sleep is real time and counts toward the speed budget (TST-47). + +### Mocks That Share State + +When several mocks stand in for one stateful dependency, such as a database connection, keep the +state in one hashtable that all of them read and change: + +```powershell +Describe 'Add-User' -Tag 'Unit', 'Public' { + BeforeAll { + $database = @{ + Connected = $false + TransactionActive = $false + Users = [System.Collections.Generic.List[object]]::new() + } + + Mock -CommandName Connect-Database -ModuleName ModuleName -MockWith { + if ($ConnectionString -match 'invalid') { + throw [System.InvalidOperationException]::new('Invalid connection string') + } + $database.Connected = $true + } + Mock -CommandName Start-DatabaseTransaction -ModuleName ModuleName -MockWith { + if (-not $database.Connected) { + throw [System.InvalidOperationException]::new('Database connection not established') + } + $database.TransactionActive = $true + } + Mock -CommandName Invoke-DatabaseQuery -ModuleName ModuleName -MockWith { + $user = [PSCustomObject]@{ + Id = $database.Users.Count + 1 + Name = $Parameters.Name + Email = $Parameters.Email + } + $database.Users.Add($user) + [PSCustomObject]@{ RowsAffected = 1; Identity = $user.Id } + } + Mock -CommandName Complete-DatabaseTransaction -ModuleName ModuleName -MockWith { + $database.TransactionActive = $false + } + } + + AfterEach { + $database.Connected = $false + $database.TransactionActive = $false + $database.Users.Clear() + } + + It 'Stores the user and closes the transaction' { + $id = Add-User -Name 'Ada' -Email 'ada@example.test' + + $id | Should-Be 1 + $database.Users | Should-BeCollection -Count 1 + $database.Users[0].Email | Should-Be 'ada@example.test' + $database.TransactionActive | Should-BeFalse + } +} +``` + +The mocks change members of the hashtable and never assign the variable, so every block works on +the same object. Pester removes the mocks when their block ends, but not the state: reset it in +`AfterEach`. Delete there, too, any temporary file that a mock created outside `$TestDrive`: + +```powershell +AfterEach { + Get-ChildItem -LiteralPath ([System.IO.Path]::GetTempPath()) -Filter 'MockTest*' | + Remove-Item -Force -ErrorAction SilentlyContinue +} +``` + ### The Clock Code that calls `Get-Date` gives a different result on every run. Fix the time (TST-31): @@ -175,9 +381,49 @@ It 'Reports the age of the report in days' { ### The File System -Prefer real files in `$TestDrive` to mocks of `Get-Content`, `Test-Path`, and `Set-Content` (TST-03). -The test then exercises the real provider, and Pester removes the files. Mock a file command only -for a condition that cannot be staged, such as a read that is denied: +Mock the file commands with realistic behaviour when a unit test is to stay off the disk. The mocks +of `Test-Path`, `Get-Content`, `Set-Content`, `New-Item`, and `Remove-Item` then share an in-memory +store, in the way that "Mocks That Share State" describes: + +```powershell +Describe 'Update-Setting' -Tag 'Unit', 'Public' { + BeforeAll { + $settingPath = 'C:\Temp\settings.txt' + $files = @{} + + Mock -CommandName Test-Path -ModuleName ModuleName -MockWith { + $files.ContainsKey([string]$LiteralPath) + } + Mock -CommandName Get-Content -ModuleName ModuleName -MockWith { + if (-not $files.ContainsKey([string]$LiteralPath)) { + throw [System.IO.FileNotFoundException]::new("File not found: $LiteralPath") + } + $files[[string]$LiteralPath] + } + Mock -CommandName Set-Content -ModuleName ModuleName -MockWith { + $files[[string]$LiteralPath] = $Value -join "`r`n" + } + } + + BeforeEach { + $files.Clear() + $files[$settingPath] = 'region=westeurope' + } + + It 'Replaces the content of an existing settings file' { + Update-Setting -LiteralPath $settingPath -Value 'region=northeurope' + + $files[$settingPath] | Should-Be 'region=northeurope' + Should-Invoke -CommandName Set-Content -ModuleName ModuleName -Times 1 -Exactly + } +} +``` + +`-LiteralPath` reaches the mock as a `[string[]]`, so the mock converts it before it uses it as a +key. + +A test that is to exercise the real provider creates real files in `$TestDrive` (TST-03), and Pester +removes them. A condition that cannot be staged there, such as a read that is denied, is mocked: ```powershell It 'Reports a settings file that cannot be read' { @@ -220,6 +466,75 @@ It 'Returns the changed files' { Set the exit code in every mock of an executable, the successful ones too. The variable keeps the value that the previous test left in it. +Code that starts the program with `Start-Process` and reads the exit code of the process object is +tested by mocking `Start-Process`. The mock returns an object with an `ExitCode`, and a `switch` on +the command line gives each command its own: + +```powershell +BeforeAll { + Mock -CommandName Start-Process -ModuleName ModuleName -MockWith { + switch -Regex ("$FilePath $ArgumentList") { + 'git status' { + if ($PesterBoundParameters.ContainsKey('RedirectStandardOutput')) { + Set-Content -LiteralPath $RedirectStandardOutput -Value ' M README.md' + } + return [PSCustomObject]@{ ExitCode = 0; ProcessName = 'git' } + } + 'git clone.*invalid-repo' { + return [PSCustomObject]@{ ExitCode = 128; ProcessName = 'git' } + } + default { + return [PSCustomObject]@{ ExitCode = 127; ProcessName = $FilePath } + } + } + } +} + +It 'Fails when git reports that the repository does not exist' { + { Copy-Repository -Uri 'https://example.test/invalid-repo' -Destination $TestDrive } | + Should-Throw -FullyQualifiedErrorId 'GitCloneFailed,Copy-Repository' +} +``` + +The mock writes the output that the program would print to the file named by +`-RedirectStandardOutput`, when the caller passed that parameter. It asks `$PesterBoundParameters` +first: a mock body runs under the strict mode of the module that calls it, where reading the +variable of a parameter that was not passed is an error. + +### A Command With Typed Parameters + +PowerShell binds the arguments of a mocked command with the parameter types of the real command, +before the mock body runs. A stand-in object of another type is rejected: + +```text +Cannot process argument transformation on parameter 'CimSession'. Cannot convert the +"@{ComputerName=System.String[]}" value of type "System.Management.Automation.PSCustomObject" to +type "Microsoft.Management.Infrastructure.CimSession[]". +``` + +`-RemoveParameterType` takes the type off the named parameter of the mock, so that the stand-in +session of a CIM test is accepted: + +```powershell +BeforeEach { + Mock -CommandName New-CimSession -ModuleName ModuleName -MockWith { + [PSCustomObject]@{ ComputerName = $ComputerName } + } + Mock -CommandName Remove-CimSession -ModuleName ModuleName -RemoveParameterType CimSession -MockWith { } + Mock -CommandName Get-CimInstance -ModuleName ModuleName -RemoveParameterType CimSession -MockWith { + [PSCustomObject]@{ FreePhysicalMemory = 4GB } + } +} + +It 'Reads the free memory through the session and closes the session' { + $computerName = 'server01.example.test' + + Get-FreeMemory -ComputerName $computerName | Should-Be 4GB + + Should-Invoke -CommandName Remove-CimSession -ModuleName ModuleName -Times 1 -Exactly +} +``` + ### Running The Real Command From A Mock To count the calls to a command that should still do its work, or to run the real command for some @@ -268,6 +583,8 @@ It 'Sends one GET request for each name' { - `-Times 1` alone means at least once. Add `-Exactly` (TST-22). Without `-Times`, `Should-Invoke` passes for one call or more. +- `-Exactly` is a switch. `Should-Invoke Cmd -Exactly 1` binds `1` to `-Times` by position and so + happens to work, but `-Times 1 -Exactly` is clearer and matches the classic syntax. - `-ExclusiveFilter` passes only when every recorded call matches the filter. Use it to prove that nothing else was requested: @@ -365,6 +682,9 @@ BeforeAll { - The dot-sourcing must be inside `BeforeAll`. Top-level code in `Pester.BeforeContainer.ps1` runs during discovery only, so a function defined there no longer exists when the tests run. - The mock belongs to the block that calls the function, as if the block had declared it. +- `Mock` itself must still run in a setup block or a test, so call the function there. Called at the + top level of a file or directly in the body of a `Describe`, it runs during discovery: Pester + 6.2.0 reports no error, and the command is not mocked when the tests run. - Pester's source marks `Pester.BeforeContainer.ps1` as experimental. ## Global Mocks (Experimental) @@ -383,6 +703,8 @@ $configuration.Mock.Global = $true | Reach of a mock | The test file, or the module named by `-ModuleName` | Every module and script | | `-ModuleName` | Selects the caller that the mock applies to | Still accepted; it no longer limits the mock | | A call that matches no filter | Error, as described above | Error, as described above | +| Lifetime of a mock | Until the declaring block ends | Unchanged | +| A nested Pester run | Does not get the mock. A call from the scope that holds it fails with `No mock for command ... is defined in this scope` | Does not get the mock, which is tied to the run that created it. The command runs for real | Its use is a guard: a mock that fails the test when any code, in any module, runs a command for real. A guard that has a filter needs a default mock beside it, because the calls that the guard @@ -404,3 +726,10 @@ The filter reads both path parameters, because the code under test may use eithe off, the same two mocks apply to calls from the test file only. A module that calls `Remove-Item` runs the real command, and the guard never fires. To let the permitted calls delete for real, make the default mock call the real command as shown under "Running The Real Command From A Mock". + +`Mock.Global` is experimental and may change. The Pester team has said that it wants this to become +the default in Pester 7, and it collects reports either way, including "turned it on, nothing +changed", through the Pester issue tracker. + +Configure it for a run, not for a file. There is no file-level directive to opt in or out, so a +suite either runs with global mocks or it does not. diff --git a/.github/instructions/pester-supporting-docs/performance-test-template.md b/.github/instructions/pester-supporting-docs/performance-test-template.md index d99c75b..38f9b60 100644 --- a/.github/instructions/pester-supporting-docs/performance-test-template.md +++ b/.github/instructions/pester-supporting-docs/performance-test-template.md @@ -1,9 +1,13 @@ # Performance Test Template -A test file for a function that has a stated performance requirement. Write one only for such a -function (TST-43), and take the limits from the requirement; the figures below are placeholders. -The rules are in the [Pester rules](../powershell-pester.instructions.md). The file below passes on -Pester 6.2.0 under PowerShell 7.6 and Windows PowerShell 5.1. +Use this template for performance benchmarking and regression detection. TST-43 requires +performance tests for a function that processes more than 100 items; they validate timing and +memory use. The limits come from a baseline file and from the requirement; the figures below are +placeholders. The rules are in the [Pester rules](../powershell-pester.instructions.md). The file +below passes on Pester 6.2.0 under PowerShell 7.6 and Windows PowerShell 5.1. + +Do not use emoji in generated code, documentation, or test output. Use plain text and standard +ASCII characters only (STY-06, TST-07). ## The Code Under Test @@ -11,57 +15,337 @@ The template tests the module `ModuleName` from the [unit test template](./unit- `Get-ThingSummary -Thing ` reads each thing once and emits one object for each state, with the number of things in that state. +## The Baseline File + +The limits that change as the code changes are kept in a data file, so that a new baseline is a +change to data and not to the tests. A test falls back to a default when the file, or a figure in +it, is missing. + +```powershell +# Tests/TestData/Baselines/performance-baselines.psd1 +@{ + BaselineInputMilliseconds = 1000 +} +``` + +A `.psd1` file holds literals only, so a duration is written as a number of milliseconds, not as a +`[timespan]`. + ## The Test File ```powershell #Requires -Modules @{ ModuleName = 'Pester'; ModuleVersion = '6.2.0' } #pester:no-parallel +#pester:no-shuffle BeforeAll { $moduleRoot = Split-Path -Path $PSScriptRoot -Parent + $modulePath = Join-Path -Path $moduleRoot -ChildPath 'ModuleName.psd1' if (-not (Get-Module -Name ModuleName)) { - Import-Module -Name (Join-Path -Path $moduleRoot -ChildPath 'ModuleName.psd1') + Import-Module -Name $modulePath + } + + # Performance test configuration + $performanceConfig = @{ + BaselineDataSize = 2000 + ScaleTestDataSize = 20000 + MaxExecutionTime = [timespan]::FromSeconds(5) + MaxMemoryUsageMB = 100 + WarmupIterations = 3 + TestIterations = 5 } - $largeInput = foreach ($index in 1..20000) { + # Load the performance baselines. Without the file, each test falls back to its default. + $dataPath = Join-Path -Path $PSScriptRoot -ChildPath 'TestData' + $baselineFolder = Join-Path -Path $dataPath -ChildPath 'Baselines' + $baselinePath = Join-Path -Path $baselineFolder -ChildPath 'performance-baselines.psd1' + if (Test-Path -LiteralPath $baselinePath) { + $performanceBaselines = Import-PowerShellDataFile -LiteralPath $baselinePath + } else { + $performanceBaselines = @{} + } + + $largeInput = foreach ($index in 1..$performanceConfig.ScaleTestDataSize) { $state = if ($index % 3 -eq 0) { 'failed' } elseif ($index % 2 -eq 0) { 'stopped' } else { 'ready' } [PSCustomObject]@{ Name = "thing-$index"; State = $state } } - $smallInput = $largeInput | Select-Object -First 2000 + $smallInput = $largeInput | Select-Object -First $performanceConfig.BaselineDataSize + + # The tests add their figures here, and AfterAll saves them. + $performanceMetrics = @{} # The first call pays for module loading and compilation, so it is made before any measurement. $null = Get-ThingSummary -Thing $smallInput } -Describe 'Get-ThingSummary performance' -Tag 'Performance' { - It 'Summarizes 2,000 things in less than one second' { - { Get-ThingSummary -Thing $smallInput } | Should-BeFasterThan '1s' +AfterAll { + # Save the performance metrics for comparison with later runs. + $resultsPath = Join-Path -Path $PSScriptRoot -ChildPath 'Results' + if (-not (Test-Path -LiteralPath $resultsPath)) { + $null = New-Item -Path $resultsPath -ItemType Directory } - It 'Summarizes 20,000 things correctly in less than five seconds' { - $elapsed = Measure-Command { - $result = @(Get-ThingSummary -Thing $largeInput) + $performanceMetrics.TestDate = (Get-Date).ToString('o') + $performanceMetrics.PowerShellVersion = $PSVersionTable.PSVersion.ToString() + $performanceMetrics.Platform = [System.Environment]::OSVersion.Platform.ToString() + + $fileName = 'performance-results-{0}.json' -f (Get-Date -Format 'yyyyMMdd-HHmmss') + $json = ConvertTo-Json -InputObject $performanceMetrics -Depth 10 + Set-Content -LiteralPath (Join-Path -Path $resultsPath -ChildPath $fileName) -Value $json -Encoding ascii +} + +Describe 'Get-ThingSummary performance' -Tag 'Performance', 'Benchmark' { + Context 'Execution time baselines' { + BeforeEach { + # Warm up the function. + foreach ($iteration in 1..$performanceConfig.WarmupIterations) { + $null = Get-ThingSummary -Thing $smallInput + } + + # Force garbage collection for a consistent baseline. + [System.GC]::Collect() + [System.GC]::WaitForPendingFinalizers() + } + + It 'Should process the baseline input faster than the limit' { + { Get-ThingSummary -Thing $smallInput } | Should-BeFasterThan '1s' + } + + It 'Should stay within the baseline across iterations' { + $baselineMs = 1000 + if ($performanceBaselines.ContainsKey('BaselineInputMilliseconds')) { + $baselineMs = $performanceBaselines.BaselineInputMilliseconds + } + + # Run several iterations for an accurate measurement. + $measurements = foreach ($iteration in 1..$performanceConfig.TestIterations) { + (Measure-Command { $null = Get-ThingSummary -Thing $smallInput }).TotalMilliseconds + } + $averageMs = ($measurements | Measure-Object -Average).Average + $maximumMs = ($measurements | Measure-Object -Maximum).Maximum + $performanceMetrics.AverageMilliseconds = $averageMs + $performanceMetrics.MaximumMilliseconds = $maximumMs + + $averageMs | Should-BeLessThan $baselineMs + $maximumMs | Should-BeLessThan ($baselineMs * 1.5) -Because 'the peak may exceed the baseline by half' + } + + It 'Should scale efficiently with data volume' { + $smallRuns = foreach ($iteration in 1..$performanceConfig.TestIterations) { + (Measure-Command { $null = Get-ThingSummary -Thing $smallInput }).TotalMilliseconds + } + $largeRuns = foreach ($iteration in 1..$performanceConfig.TestIterations) { + (Measure-Command { $null = Get-ThingSummary -Thing $largeInput }).TotalMilliseconds + } + $smallBest = ($smallRuns | Measure-Object -Minimum).Minimum + $largeBest = ($largeRuns | Measure-Object -Minimum).Minimum + + # A floor of one millisecond keeps timer resolution from deciding the ratio. + $dataScaleFactor = $performanceConfig.ScaleTestDataSize / $performanceConfig.BaselineDataSize + $timeScaleFactor = $largeBest / [math]::Max($smallBest, 1) + $scalingEfficiency = $timeScaleFactor / $dataScaleFactor + $performanceMetrics.ScalingEfficiency = $scalingEfficiency + + $scalingEfficiency | Should-BeLessThan 1.5 -Because 'the summary reads each thing once' } - $result | Should-BeCollection -Count 3 - ($result | Measure-Object -Property Count -Sum).Sum | Should-Be 20000 - $elapsed.TotalMilliseconds | Should-BeLessThan 5000 + It 'Should complete a large data set within the time limit' { + $elapsed = Measure-Command { + $result = @(Get-ThingSummary -Thing $largeInput) + } + + $result | Should-BeCollection -Count 3 + ($result | Measure-Object -Property Count -Sum).Sum | Should-Be $performanceConfig.ScaleTestDataSize + $elapsed.TotalMilliseconds | Should-BeLessThan $performanceConfig.MaxExecutionTime.TotalMilliseconds + } + + It 'Should maintain consistent performance across iterations' { + # One measurement repeats the call until it lasts about a quarter of a second. + $single = (Measure-Command { $null = Get-ThingSummary -Thing $largeInput }).TotalMilliseconds + $callCount = [int][math]::Ceiling(250 / [math]::Max($single, 1)) + $iterations = 10 + + $measurements = foreach ($iteration in 1..$iterations) { + [System.GC]::Collect() + [System.GC]::WaitForPendingFinalizers() + $elapsed = Measure-Command { + foreach ($call in 1..$callCount) { + $null = Get-ThingSummary -Thing $largeInput + } + } + $elapsed.TotalMilliseconds + } + + # Coefficient of variation: the standard deviation as a share of the average. + $average = ($measurements | Measure-Object -Average).Average + $squares = foreach ($measurement in $measurements) { + ($measurement - $average) * ($measurement - $average) + } + $standardDeviation = [math]::Sqrt(($squares | Measure-Object -Sum).Sum / $iterations) + $variation = $standardDeviation / $average + $performanceMetrics.CoefficientOfVariation = $variation + + $variation | Should-BeLessThan 0.20 -Because 'the time should be stable across runs' + } } - It 'Takes less than 30 times as long for 10 times the input' { - $smallRuns = foreach ($run in 1..5) { - (Measure-Command { $null = Get-ThingSummary -Thing $smallInput }).TotalMilliseconds + Context 'Memory usage' { + It 'Should maintain a reasonable memory footprint' { + $beforeMemory = [System.GC]::GetTotalMemory($true) + + $null = Get-ThingSummary -Thing $largeInput + + # No collection is forced here, so the figure includes what the call left behind. + $afterMemory = [System.GC]::GetTotalMemory($false) + $memoryUsedMB = ($afterMemory - $beforeMemory) / 1MB + $performanceMetrics.MemoryUsedMB = $memoryUsedMB + + $memoryUsedMB | Should-BeLessThan $performanceConfig.MaxMemoryUsageMB } - $largeRuns = foreach ($run in 1..5) { - (Measure-Command { $null = Get-ThingSummary -Thing $largeInput }).TotalMilliseconds + + It 'Should release memory after completion' { + $beforeMemory = [System.GC]::GetTotalMemory($true) + + $null = Get-ThingSummary -Thing $largeInput + + [System.GC]::Collect() + [System.GC]::WaitForPendingFinalizers() + [System.GC]::Collect() + $afterMemory = [System.GC]::GetTotalMemory($true) + $memoryDifferenceMB = [math]::Abs($afterMemory - $beforeMemory) / 1MB + + # Most of the memory must be released. The tolerance is 10 MB. + $memoryDifferenceMB | Should-BeLessThan 10 -Because 'the function should not retain allocations' } - $smallBest = ($smallRuns | Measure-Object -Minimum).Minimum - $largeBest = ($largeRuns | Measure-Object -Minimum).Minimum - # A floor of one millisecond keeps timer resolution from deciding the ratio. - $ratio = $largeBest / [math]::Max($smallBest, 1) - $ratio | Should-BeLessThan 30 -Because 'the summary reads each thing once' + It 'Should not have memory leaks with repeated calls' { + $initialMemory = [System.GC]::GetTotalMemory($true) + + # Measure the memory after every fifth call. + $memoryMeasurements = foreach ($call in 1..20) { + $null = Get-ThingSummary -Thing $smallInput + if ($call % 5 -eq 0) { + [System.GC]::Collect() + ([System.GC]::GetTotalMemory($true) - $initialMemory) / 1MB + } + } + + # Memory must not keep growing: at most 5 MB between the first and the last measurement. + $memoryMeasurements | Should-BeCollection -Count 4 + $memoryGrowth = $memoryMeasurements[-1] - $memoryMeasurements[0] + $memoryGrowth | Should-BeLessThan 5 -Because 'repeated calls must not leak' + } + } + + Context 'Throughput and scalability' { + It 'Should achieve the minimum throughput' { + # Things for each second + $requiredThroughput = 50 + + $elapsed = Measure-Command { + $result = @(Get-ThingSummary -Thing $smallInput) + } + + ($result | Measure-Object -Property Count -Sum).Sum | Should-Be $performanceConfig.BaselineDataSize + $actualThroughput = $performanceConfig.BaselineDataSize / $elapsed.TotalSeconds + $performanceMetrics.ThingsPerSecond = $actualThroughput + $actualThroughput | Should-BeGreaterThan $requiredThroughput + } + + It 'Should handle concurrent execution efficiently' { + $thingCount = $performanceConfig.ScaleTestDataSize + $runCount = $performanceConfig.TestIterations + $jobScript = { + param($ModulePath, $ThingCount, $RunCount) + + Import-Module -Name $ModulePath + $things = foreach ($index in 1..$ThingCount) { + [PSCustomObject]@{ Name = "thing-$index"; State = 'ready' } + } + $null = Get-ThingSummary -Thing $things + $runs = foreach ($run in 1..$RunCount) { + (Measure-Command { $summary = @(Get-ThingSummary -Thing $things) }).TotalMilliseconds + } + + [PSCustomObject]@{ + ItemsProcessed = ($summary | Measure-Object -Property Count -Sum).Sum + BestMilliseconds = ($runs | Measure-Object -Minimum).Minimum + } + } + + $jobs = foreach ($jobNumber in 1..5) { + Start-Job -ScriptBlock $jobScript -ArgumentList $modulePath, $thingCount, $runCount + } + $jobResults = @($jobs | Wait-Job | Receive-Job) + $jobs | Remove-Job + + # Every job completed and processed every thing. + $jobResults | Should-BeCollection -Count 5 + $jobResults | Should-All { $_.ItemsProcessed -eq $thingCount } + $jobResults | Should-All { $_.BestMilliseconds -lt 30000 } + + # A call in a job must not take more than twice as long as a call made alone. + $serialRuns = foreach ($run in 1..$runCount) { + (Measure-Command { $null = Get-ThingSummary -Thing $largeInput }).TotalMilliseconds + } + $serialBest = ($serialRuns | Measure-Object -Minimum).Minimum + $averageConcurrent = ($jobResults.BestMilliseconds | Measure-Object -Average).Average + $averageConcurrent | Should-BeLessThan ([math]::Max($serialBest, 1) * 2) + } + } + + Context 'Resource utilization' { + It 'Should use CPU resources efficiently' { + $ownSample = $null + if (Get-Command -Name Get-Counter -ErrorAction SilentlyContinue) { + $processName = (Get-Process -Id $PID).ProcessName + $idCounterPath = "\Process($processName*)\ID Process" + $idCounter = Get-Counter -Counter $idCounterPath -ErrorAction SilentlyContinue + if ($null -ne $idCounter) { + $ownSample = $idCounter.CounterSamples | + Where-Object { $_.CookedValue -eq $PID } | + Select-Object -First 1 + } + } + + if ($null -eq $ownSample) { + Set-ItResult -Skipped -Because 'CPU performance counters are not available' + } else { + $cpuCounter = $ownSample.Path -replace 'id process$', '% processor time' + $before = (Get-Counter -Counter $cpuCounter).CounterSamples[0] + + foreach ($call in 1..10) { + $null = Get-ThingSummary -Thing $largeInput + } + + $after = (Get-Counter -Counter $cpuCounter).CounterSamples[0] + + # The raw value counts processor time in the unit of the time stamp. 100 percent of + # the result is every processor in use for the whole interval. + $processorTime = $after.RawValue - $before.RawValue + $interval = ($after.TimeStamp100nSec - $before.TimeStamp100nSec) * [Environment]::ProcessorCount + $cpuUsage = 100 * $processorTime / $interval + $performanceMetrics.ProcessorPercent = $cpuUsage + + $cpuUsage | Should-BeGreaterThan 0 + $cpuUsage | Should-BeLessThan 80 + } + } + + It 'Should not create excessive temporary files' { + $tempPath = [System.IO.Path]::GetTempPath() + $startTime = Get-Date + + $null = Get-ThingSummary -Thing $largeInput + + $newFiles = @( + Get-ChildItem -LiteralPath $tempPath -File | Where-Object { $_.CreationTime -ge $startTime } + ) + + # Other processes write to this folder too, so the limit is five files and not zero. + $newFiles.Count | Should-BeLessThanOrEqual 5 + } } } ``` @@ -74,14 +358,14 @@ Describe 'Get-ThingSummary performance' -Tag 'Performance' { message. Use it for a plain limit. Pipe the script block, not its result: any input other than a script block or a `[timespan]` is an error. -Use `Measure-Command` when the test needs the number itself, for a ratio or for the best of several -runs, or when it must also assert the result of the measured call. +Use `Measure-Command` when the test needs the number itself, for a ratio, an average, or a +coefficient of variation, or when it must also assert the result of the measured call. ### Assertions Stay Outside The Measurement -An assertion inside `Measure-Command` adds its own time to the figure. The second test assigns the -result inside the measured block, where the variable stays visible afterwards, and asserts after -the measurement. Asserting the result also proves that the fast call did the work. +An assertion inside `Measure-Command` adds its own time to the figure. The large-input test assigns +the result inside the measured block, where the variable stays visible afterwards, and asserts +after the measurement. Asserting the result also proves that the fast call did the work. ### Like Is Compared With Like @@ -101,19 +385,126 @@ $elapsed | Should-BeLessThan ([timespan]::FromSeconds(5)) ### Measurements Are Made Stable - The input is built once in `BeforeAll`, and one call is made before any measurement. -- A ratio uses the best of several runs. A single run can include a garbage collection or another - process taking the processor. +- `BeforeEach` warms the function up again and forces a garbage collection, so that every timing + test starts from the same state. +- The baseline test runs several iterations and asserts the average and the maximum. The maximum + may exceed the baseline by half. +- The scaling test uses the best of several runs instead. The minimum is the better figure for a + ratio, because a single run can include a garbage collection or another process taking the + processor. - Limits leave room for a slower machine. A limit close to the usual time makes the test fail at random, and a test that fails at random gets ignored. +### Scaling + +The scaling test divides the growth in time by the growth in data. A function that reads each item +once stays near 1: measured here, between 0.5 and 1.2. The limit of 1.5 leaves room for noise and +fails a function whose time grows clearly faster than its input. + +### Consistency + +The consistency test computes the coefficient of variation over 10 iterations and requires less +than 20 percent. A measurement of a few milliseconds varies with the timer and the scheduler more +than with the code, so each measurement repeats the call until it lasts about a quarter of a +second. Measured here with this function: measurements of about 10 milliseconds gave a variation +between 0.13 and 0.33, and measurements of 250 milliseconds stayed below 0.15. + +### Memory + +Three tests cover memory: the footprint during a call, the release after it, and growth over +repeated calls. `[System.GC]::GetTotalMemory($true)` collects before it reads, which gives the +starting figure; the footprint test reads the second figure without a collection. + +### Throughput And Concurrent Runs + +The throughput test divides the number of items by the elapsed time and asserts the result of the +call as well. The concurrency test starts five jobs. A job is a new process, so the script block +receives the module path as an argument, imports the module, and makes one call before it measures. +The test then compares the time of a call in a job with the time of a call made alone. + +### Processor And Temporary Files + +The processor test reads the `% Processor Time` counter of its own process before and after the +work and computes the share from the raw values of the two samples. It finds its own counter +instance through the `ID Process` counter, because the instance name does not contain the process +ID: measured here, a filter on the ID in the instance name matched no sample, and the test then +compared nothing. The test is skipped when `Get-Counter` or the counter is not available. + +The share is of all processors together. A single-threaded call that keeps one processor busy +reads 100 divided by the number of processors; measured here on 20 processors it was 4 percent. + +The temporary-file test counts the files that appeared in the temporary folder during the call. + +### Metrics Are Saved And Compared + +Each test adds its figure to `$performanceMetrics`, and the top-level `AfterAll` writes them to a +JSON file in `Tests/Results`, one file for each run. The saved files are the history that the +regression guidelines below rely on, and `performance-baselines.psd1` holds the baseline that a +test compares with. + +The large input can also come from a shared helper; see +[Test Data Management Guide](./test-data-guide.md#performance-test-data). + ### The File Does Not Run In Parallel -Timings taken while other test files use the processor are noise. The comment `#pester:no-parallel` -keeps the file in the main session, one file at a time, when a run sets `Run.Parallel`. That option -is experimental and off by default. +Timings taken while other test files use the processor are noise. Put the comment +`#pester:no-parallel` at the top of every performance test file. It keeps the file in the main +session, one file at a time, when a run sets `Run.Parallel`. That option is experimental and off by +default. + +As a second safeguard, also keep `Run.Parallel = $false` in `PesterConfiguration.Performance.psd1`, +the settings file of the performance run; see +[Pester Configuration Guide](./pester-configuration.md#settings-for-each-environment). + +### The Run Is Not Shuffled + +A performance baseline assumes a fixed order. Do not turn `Run.Shuffle` on for a run of performance +tests. The runner script in [Test Execution Guide](./test-execution.md#the-test-runner-script) +shuffles every other run and leaves performance tests in their declared order (TST-46). + +Put the comment `#pester:no-shuffle` at the top of every performance test file as well. A run of +every test type is shuffled, and the comment keeps the blocks and tests of the file in the order +they are written; see [Test Execution Guide](./test-execution.md#shuffled-test-order). ### Coverage Is Off Do not enable code coverage in a run that measures time. Coverage instruments the code that is being -measured. Tag the `Describe` block `Performance` (TST-04), and run these tests in their own job with -`Filter.Tag`, without `CodeCoverage.Enabled`. +measured. Tag the `Describe` block `Performance` and with the category, here `Benchmark` (TST-04), +and run these tests in their own job with `Filter.Tag`, without `CodeCoverage.Enabled`. + +## Performance Test Guidelines + +### Baseline Establishment + +- Warm up functions before measurement. +- Run several iterations for accuracy. +- Force garbage collection for consistency. +- Establish realistic performance targets. + +### Memory Testing + +- Measure memory usage during execution. +- Verify memory release after completion. +- Test for memory leaks with repeated calls. +- Monitor garbage collection patterns. + +### Scalability Testing + +- Test with increasing data sizes. +- Measure throughput and response time. +- Validate concurrent execution performance. +- Test the limits of resource utilization. + +### Regression Detection + +- Compare against historical baselines. +- Track performance trends over time. +- Alert on significant performance degradation. +- Maintain a database of performance metrics. + +### Resource Monitoring + +- Monitor CPU utilization. +- Track the creation of temporary files. +- Measure network usage where it applies. +- Validate resource cleanup. diff --git a/.github/instructions/pester-supporting-docs/pester-configuration.md b/.github/instructions/pester-supporting-docs/pester-configuration.md index 775eaa2..8354344 100644 --- a/.github/instructions/pester-supporting-docs/pester-configuration.md +++ b/.github/instructions/pester-supporting-docs/pester-configuration.md @@ -6,6 +6,9 @@ easy to get wrong. It supports the testing rules in defaults, and behaviour were read from the configuration object of Pester 6.2.0 and checked by running it. `Get-Help about_PesterConfiguration` shows the same list on the installed version. +Do not use emoji in generated code, in documentation, or in test output. Use plain text and ASCII +characters only. + ## Creating a Configuration ```powershell @@ -32,30 +35,156 @@ $target = $configuration.CodeCoverage.CoveragePercentTarget.Value ### Loading Settings From a File -`New-PesterConfiguration -Hashtable` accepts a hashtable that contains only the options to change, -so settings can be kept in a `.psd1` file: +`New-PesterConfiguration -Hashtable` accepts a hashtable, so settings can be kept in a `.psd1` +file. Keep one standard file that lists every section, so that each setting and its value can be +read in one place. + +The settings files and the runner script are kept in the `Tests` folder, as the layout in +[Pester Test Structure Guide](./test-structure-guide.md#test-directory-layout) shows. A `build` +folder beside `Tests` is the alternative; both are one level below the repository root, so the +scripts are the same in either. ```powershell -# build/PesterConfiguration.psd1 +# Tests/PesterConfiguration.psd1 @{ Run = @{ + Path = @('./Tests/Unit', './Tests/Integration') + ExcludePath = @() PassThru = $true + SkipRun = $false + + # An empty -ForEach fails discovery. Keep this on, and opt out for one test with + # -AllowNullOrEmptyForEach where an empty set is expected. + FailOnNullOrEmptyForEach = $true + + # None, Run, Container, or Block + SkipRemainingOnFailure = 'None' + + # Experimental and off by default. See "Experimental Options". + Parallel = $false + ParallelThrottleLimit = 0 + + # Experimental and off by default in Pester, but on in every run here (TST-46). Only + # the performance file turns it off. A seed of 0 picks a new seed for each run and + # prints it. + Shuffle = $true + ShuffleSeed = 0 + } + + Output = @{ + Verbosity = 'Detailed' + + # None, FirstLine, Filtered, or Full + StackTraceVerbosity = 'Filtered' + + # None, Auto, AzureDevops, or GithubActions + CIFormat = 'Auto' + CILogLevel = 'Error' + + # Auto or None. Auto shows verbose and debug output when the CI system runs in debug mode. + CIDebugOutput = 'Auto' + + # Auto, Ansi, ConsoleColor, or Plaintext + RenderMode = 'Auto' + + # $true adds [Tags: ...] to each output line. + ShowTags = $false + } + + Mock = @{ + # Experimental and off by default. See "Experimental Options". + Global = $false } CodeCoverage = @{ Enabled = $true + # List only folders that exist. One entry in a folder that does not exist makes Pester + # measure nothing: remove './Classes/*.ps1' when the module has no Classes folder. + Path = @('./Public/*.ps1', './Private/*.ps1', './Classes/*.ps1') + + # JaCoCo or Cobertura + OutputFormat = 'JaCoCo' + OutputPath = './Tests/Results/Coverage.xml' + OutputEncoding = 'UTF8' + + # Not "Threshold": that option does not exist. CoveragePercentTarget = 80 + + # $false measures with the profiler-based tracer, the default in Pester 6. + UseBreakpoints = $false + SingleHitBreakpoints = $true + + # Keeps test files out of the coverage figures. + ExcludeTests = $true + RecursePaths = $true } TestResult = @{ Enabled = $true + + # NUnitXml, NUnit2.5, NUnit3, or JUnitXml OutputFormat = 'NUnitXml' + OutputPath = './Tests/Results/TestResults.xml' + OutputEncoding = 'UTF8' + TestSuiteName = 'PowerShell Tests' + } + + Should = @{ + # 'Continue' reports every failed assertion of a test instead of the first one. + ErrorAction = 'Stop' + + # $true makes the Should -Be style fail. + DisableV5 = $false + } + + Debug = @{ + ShowFullErrors = $false + WriteDebugMessages = $false + ShowNavigationMarkers = $false + + # Writes a line when each test starts. + ShowStartMarkers = $false + ReturnRawResultObject = $false + } + + Filter = @{ + Tag = @() + ExcludeTag = @() + Line = @() + ExcludeLine = @() + FullName = @() } } ``` -Paths in a configuration are relative to the current location, which a build script does not -control. Keep paths out of the file and set them in the script from `$PSScriptRoot`: +`CodeCoverage.ReportRoot` is left out of the file: when it is not set, `Run.RepoRoot` is used. +`Run.Throw` and `Run.Exit` are left out as well; the reason is under the [Run](#run) options. + +The paths in the file are relative to the current location when the run starts, which a build +script does not control. Measured on 6.2.0 with `Run.Path = './Tests/Unit'`: a run started in the +repository root found the tests, and a run started in another folder failed with +`No test files were found`. The script that loads the file therefore finds the repository root from +`$PSScriptRoot` and runs Pester from there: + +```powershell +$repositoryRoot = Split-Path -Path $PSScriptRoot -Parent +$settingsPath = Join-Path -Path $PSScriptRoot -ChildPath 'PesterConfiguration.psd1' +$settings = Import-PowerShellDataFile -LiteralPath $settingsPath + +$configuration = New-PesterConfiguration -Hashtable $settings +$configuration.Run.RepoRoot = $repositoryRoot + +Push-Location -LiteralPath $repositoryRoot +try { + $result = Invoke-Pester -Configuration $configuration +} finally { + Pop-Location +} +``` + +The alternative is a file that holds only the options to change and no paths, with the paths set +in the script from `$PSScriptRoot`. The CI gate script uses this form; see +[CI/CD Integration Guide](./cicd-integration.md#the-gate-script). ```powershell $repositoryRoot = Split-Path -Path $PSScriptRoot -Parent @@ -71,9 +200,205 @@ $configuration.CodeCoverage.Path = @( ) ``` -To use different settings for one kind of run, load the same file and then assign the options that -differ. The coverage figure that CI enforces is 80 percent (TST-40); do not keep a second, lower -figure for other runs. +### Settings for Each Environment + +Keep one file for each environment beside the standard file. Each holds only the settings that +differ, and is merged over the standard file when the configuration is loaded. + +```powershell +# Tests/PesterConfiguration.Development.psd1 +@{ + Run = @{ + Path = @('./Tests/Unit') + PassThru = $true + } + + Output = @{ + Verbosity = 'Detailed' + } + + CodeCoverage = @{ + Enabled = $true + + # A lower target for development runs. + CoveragePercentTarget = 70 + } +} +``` + +The target only sets the figure that Pester reports against +([The Target Is Not a Gate](#the-target-is-not-a-gate)), so the lower development figure does not +change what CI enforces: the CI file sets 80 (TST-40). The alternative is one figure, 80, in every +file. + +```powershell +# Tests/PesterConfiguration.CI.psd1 +@{ + Run = @{ + Path = @('./Tests/Unit', './Tests/Integration', './Tests/Security') + PassThru = $true + } + + Output = @{ + Verbosity = 'Normal' + + # Or 'AzureDevops'. + CIFormat = 'GithubActions' + RenderMode = 'Ansi' + } + + CodeCoverage = @{ + Enabled = $true + CoveragePercentTarget = 80 + OutputFormat = 'JaCoCo' + OutputPath = './Tests/Results/Coverage.xml' + } + + TestResult = @{ + Enabled = $true + OutputFormat = 'NUnitXml' + OutputPath = './Tests/Results/TestResults.xml' + } +} +``` + +`Auto` detects the CI system, but setting `CIFormat` in the CI file states which one the file is +for. The alternative is to leave it at `Auto`. + +```powershell +# Tests/PesterConfiguration.Performance.psd1 +@{ + Run = @{ + Path = @('./Tests/Performance') + PassThru = $true + + # Performance tests must not share the machine with other work. + Parallel = $false + + # A baseline assumes the declared order (TST-46). + Shuffle = $false + } + + Output = @{ + Verbosity = 'Normal' + } + + Filter = @{ + Tag = @('Performance', 'Benchmark') + } + + TestResult = @{ + Enabled = $true + OutputPath = './Tests/Results/PerformanceResults.xml' + } +} +``` + +### Loading and Merging the Files + +The loader reads the standard file, merges the file of the environment over it when that file +exists, and creates the configuration. The functions are kept in a helper file that the runner +script dot-sources; see [Test Execution Guide](./test-execution.md#the-test-runner-script). + +```powershell +# Tests/TestRunnerHelpers.ps1 +function Merge-Hashtable { + [CmdletBinding()] + [OutputType([hashtable])] + param( + [Parameter(Mandatory)] + [hashtable]$Base, + + [Parameter(Mandatory)] + [hashtable]$Override + ) + + $merged = $Base.Clone() + foreach ($key in $Override.Keys) { + $baseValue = $merged[$key] + $overrideValue = $Override[$key] + if ($baseValue -is [hashtable] -and $overrideValue -is [hashtable]) { + $merged[$key] = Merge-Hashtable -Base $baseValue -Override $overrideValue + } else { + $merged[$key] = $overrideValue + } + } + + $merged +} + + +function Get-PesterConfiguration { + [CmdletBinding()] + [OutputType('PesterConfiguration')] + param( + [ValidateSet('Development', 'CI', 'Production', 'Performance', 'Security', 'Integration')] + [string]$Environment = 'Development', + + [string]$ConfigPath = $PSScriptRoot + ) + + try { + $basePath = Join-Path -Path $ConfigPath -ChildPath 'PesterConfiguration.psd1' + $settings = Import-PowerShellDataFile -LiteralPath $basePath -ErrorAction Stop + + $environmentPath = Join-Path -Path $ConfigPath -ChildPath "PesterConfiguration.$Environment.psd1" + if (Test-Path -LiteralPath $environmentPath) { + # The file of the environment overrides the standard file. + $overrides = Import-PowerShellDataFile -LiteralPath $environmentPath -ErrorAction Stop + $settings = Merge-Hashtable -Base $settings -Override $overrides + } + + $configuration = New-PesterConfiguration -Hashtable $settings -ErrorAction Stop + } catch { + $PSCmdlet.ThrowTerminatingError($_) + } + + $configuration +} +``` + +A script chooses the environment from the variables that CI systems set: + +```powershell +if ($env:CI -eq 'true' -or $env:BUILD_ID) { + $environment = 'CI' +} elseif ($env:PERFORMANCE_TESTING -eq 'true') { + $environment = 'Performance' +} else { + $environment = 'Development' +} + +$configuration = Get-PesterConfiguration -Environment $environment +``` + +The same merge gives settings for an organisation with overrides for one project: + +```powershell +# Settings for every project of the organisation +$organisationSettings = @{ + CodeCoverage = @{ + CoveragePercentTarget = 80 + OutputFormat = 'JaCoCo' + } + TestResult = @{ + OutputFormat = 'NUnitXml' + } +} + +# A higher target for one critical project +$projectSettings = @{ + CodeCoverage = @{ + CoveragePercentTarget = 85 + } +} + +$settings = Merge-Hashtable -Base $organisationSettings -Override $projectSettings +$configuration = New-PesterConfiguration -Hashtable $settings +``` + +The alternative to separate files is to load the standard file and then assign the options that +differ for one kind of run. ### Checking What Was Loaded @@ -114,6 +439,72 @@ and returns no result. Test the result for `$null`, as the CI gate does. Before 6.2 a misspelled key and a value of the wrong type were both ignored without a message. +### Validating a Configuration Before the Run + +Check a loaded configuration before the run starts: that the Pester version is the pinned one, that +the test paths exist, that the folder for the reports exists, that the coverage target is a +percentage, and that no key was misspelled. The function reports what it finds and changes +nothing; the script that calls it throws. + +```powershell +# Tests/TestRunnerHelpers.ps1 +function Test-PesterConfiguration { + [CmdletBinding()] + [OutputType('TestRunner.Check')] + param( + [Parameter(Mandatory)] + [PesterConfiguration]$Configuration + ) + + $pesterVersion = (Get-Module -Name Pester).Version + $outputFolder = Split-Path -Path $Configuration.TestResult.OutputPath.Value -Parent + $target = $Configuration.CodeCoverage.CoveragePercentTarget.Value + $unknownKeys = @($Configuration.GetUnknownKeys()) + + $issues = @( + if ($pesterVersion -ne [version]'6.2.0') { + "Pester 6.2.0 is required, and $pesterVersion is loaded." + } + + foreach ($testPath in $Configuration.Run.Path.Value) { + if (-not (Test-Path -Path $testPath)) { + "Test path not found: $testPath" + } + } + + if ($outputFolder -and -not (Test-Path -LiteralPath $outputFolder)) { + "Output folder not found: $outputFolder" + } + + if ($target -gt 100 -or $target -lt 0) { + "Invalid coverage target: $target" + } + + # Invoke-Pester only warns about a key that matches no option. + if ($unknownKeys.Count -gt 0) { + "Configuration keys that match no option: $($unknownKeys -join ', ')" + } + ) + + [PSCustomObject]@{ + PSTypeName = 'TestRunner.Check' + Valid = $issues.Count -eq 0 + Issues = $issues + } +} +``` + +```powershell +$check = Test-PesterConfiguration -Configuration $configuration +if (-not $check.Valid) { + throw "The Pester configuration is not valid. $($check.Issues -join ' ')" +} +``` + +`Test-Path -Path` is used for the test paths because `Run.Path` may hold wildcards. The relative +paths of the standard file are tested against the current location, so call the function after the +script has changed to the repository root. + ## Option Reference ### Run @@ -123,7 +514,7 @@ Before 6.2 a misspelled key and a value of the wrong type were both ignored with | `Run.Path` | `.` | Folders or files to search for tests. Wildcards are allowed. | | `Run.ExcludePath` | empty | Folders or files to leave out. Each entry is compared with full paths; see [Pester Test Structure Guide](./test-structure-guide.md#hidden-folders-and-excluded-paths). | | `Run.ScriptBlock` | empty | Script blocks that contain tests. | -| `Run.Container` | empty | Containers made by `New-PesterContainer`, used to pass data to a test file. | +| `Run.Container` | empty | Containers made by `New-PesterContainer`, used to pass data to a test file. It is not a way to run files in parallel. | | `Run.TestExtension` | `.Tests.ps1` | Suffix that identifies a test file. | | `Run.PassThru` | `$false` | Return the result object. | | `Run.SkipRun` | `$false` | Discover the tests but do not run them. | @@ -195,7 +586,7 @@ them. The reserved value `None` is explained in | `Output.CIFormat` | `Auto` | Format of failures in a build log: `None`, `Auto`, `AzureDevops`, or `GithubActions`. | | `Output.CILogLevel` | `Error` | Level that failures are logged at in a build log: `Error` or `Warning`. | | `Output.CIDebugOutput` | `Auto` | `Auto` shows verbose and debug output when the CI system runs in debug mode; `None` does not. | -| `Output.RenderMode` | `Auto` | `Auto`, `Ansi`, `ConsoleColor`, or `Plaintext`. | +| `Output.RenderMode` | `Auto` | `Auto`, `Ansi`, `ConsoleColor`, or `Plaintext`; see [Render Mode](#render-mode). | | `Output.ShowTags` | `$false` | Add the tags of each block and test to its output line. | ### Should, Mock, TestDrive, and TestRegistry @@ -240,6 +631,10 @@ unknown key, and a value from the second table is rejected when the run starts. | `CodeCoverage.OutputFormat = 'CoverageGutters'` | `JaCoCo` | | `Output.Verbosity = 'Minimal'` | `Normal`. Pester accepts `Minimal` and changes it to `Normal`. | +`Run.ExcludePath` is not a substitute for `CodeCoverage.ExcludePath`. Measured on 6.2.0, a source +file or folder named in `Run.ExcludePath` by its full path, or matched by a wildcard, was still +measured: the number of analyzed files and commands did not change. + ## Reports Turn On by Themselves Setting any option of `CodeCoverage` or `TestResult` to a value other than its default turns that @@ -299,29 +694,49 @@ that match `Run.TestExtension` are left out of the figures. `CodeCoverage.CoveragePercentTarget` changes what Pester prints and writes to the report. Measured on 6.2.0, a run with 60 percent coverage and a target of 99 has the result `Passed`. The script -compares the figure itself (TST-40): +enforces the gate itself: it compares the figure with the configured target, which the result +object repeats, and lists the uncovered commands for each file when the gate fails. ```powershell $result = Invoke-Pester -Configuration $configuration $coverage = [math]::Round($result.CodeCoverage.CoveragePercent, 2) -if ($coverage -lt 80) { - throw "Code coverage is $coverage percent. The minimum is 80." +$target = $result.CodeCoverage.CoveragePercentTarget +if ($coverage -lt $target) { + $uncovered = $result.CodeCoverage.CommandsMissed | Group-Object -Property File + foreach ($file in $uncovered) { + Write-Host (' {0}: {1} uncovered command(s)' -f $file.Name, $file.Count) + } + throw "Code coverage is $coverage percent. The target is $target." } ``` +The configured target in CI is 80 (TST-40). The gate in the testing rules compares with the literal +`80` instead of the configured target; with a target of 80 the two are the same check. + ### Profiler and Breakpoints By default Pester 6 measures coverage with a profiler-based tracer. Pester 5 set a breakpoint on -every command by default. `CodeCoverage.UseBreakpoints = $true` selects the breakpoint method. +every command by default. `CodeCoverage.UseBreakpoints = $true` selects the breakpoint method; use +it only when something depends on breakpoint-based figures. + +The tracer is often described as much faster than breakpoints on a large code base. That was not +reproduced on 6.2.0. Measured on a generated code base of 60 files and 33,600 commands, on a machine +that was also doing other work: with 2,400 tests that ran 93 percent of the commands, a run took 34 +seconds without coverage, 153 and 203 seconds with the tracer, and 144 and 158 seconds with +breakpoints. With 120 tests that ran 5 percent of the commands, it took 126 and 132 seconds with the +tracer, and 137 and 152 seconds with breakpoints. Measure on the project's own code before choosing +a method for its speed. A parallel run always measures with breakpoints, whatever the option says; see [Test Execution Guide](./test-execution.md#coverage-in-a-parallel-run). ### Report Format and Paths -`JaCoCo` is the default format and `Cobertura` is the alternative. The `CoverageGutters` format of -Pester 5 was removed. +`JaCoCo` is the default format and `Cobertura` is the alternative, for GitLab, Codecov, and similar +tools. The `CoverageGutters` format of Pester 5 was removed. It existed only to write paths relative +to the repository root, and every coverage report is now written that way, so the plain `JaCoCo` +format works with the Coverage Gutters extension. Paths in the report are relative to `CodeCoverage.ReportRoot`, which defaults to `Run.RepoRoot`. Measured on 6.2.0 for the file `C/src/Code.ps1` in a repository: the JaCoCo package is `C/src` by @@ -330,8 +745,26 @@ the report expects paths relative to another folder. ## Output -`Output.Verbosity` has four values. `Normal` prints one line for each test file and the failures, -`Detailed` prints every block and test, and `Diagnostic` adds Pester's debug messages. +`Output.Verbosity` has four values. `None` prints nothing, `Normal` prints one line for each test +file and the failures, `Detailed` prints every block and test, and `Diagnostic` adds Pester's debug +messages. Use `Normal` in CI and `Detailed` in a local run. + +### Render Mode + +| `Output.RenderMode` | Meaning | +| --- | --- | +| `Auto` | The default. Detects what the terminal can do. | +| `Ansi` | Forces ANSI (VT) escape sequences. | +| `ConsoleColor` | Uses the console colours of the host and writes no escape sequences. | +| `Plaintext` | No colour. The best choice when the log is captured. | + +`Auto` becomes `Plaintext` when the `NO_COLOR` environment variable is set, `Ansi` when the host +supports virtual terminal sequences, and `ConsoleColor` otherwise. Measured on 6.2.0 with the output +of a `pwsh` process captured as text: `Auto` and `Ansi` wrote escape sequences into the text, and +`ConsoleColor` and `Plaintext` wrote none. The CI file sets `Ansi`, for a build log that shows +colour. + +### CI Output `Output.CIFormat = 'Auto'` detects Azure Pipelines (`TF_BUILD`) and GitHub Actions (`GITHUB_ACTIONS`) from their environment variables. In GitHub Actions a failed test is written as @@ -378,7 +811,8 @@ incomplete. Remove ANSI colour sequences from text before a test throws it or us `Run.Parallel`, `Run.Shuffle`, and `Mock.Global` are off by default, and Pester marks them as experimental. Do not enable `Run.Parallel` or `Mock.Global` in a configuration that you generate -unless the user asks. `Run.Shuffle` is the exception: turn it on for the unit tests in CI (TST-46). +unless the user asks. `Run.Shuffle` is the exception: turn it on in every run except a run of +performance tests (TST-46). - `Run.Parallel` and `Run.ParallelThrottleLimit`: see [Test Execution Guide](./test-execution.md#parallel-test-execution). @@ -387,3 +821,40 @@ unless the user asks. `Run.Shuffle` is the exception: turn it on for the unit te - `Mock.Global`: with the option on, a mock applies to calls of the command from any module or script, and `-ModuleName` no longer limits where it applies. See [Mocking Patterns Guide](./mocking-patterns.md). + +### Shuffle in Every Run + +Turn shuffling on in every run, on a developer's computer and in CI, with a new seed for each run +and the seed named in the failure message. A developer then meets an order dependence while +writing the test, not in the log of a CI job. The first shuffled run of one suite of 1,300 tests +failed three tests under seed 1 that file order had always hidden: each wrote into a folder that an +earlier test in the same file had created. Two other seeds passed. The same suite later moved to +importing the module once for each run instead of in every file, which makes state visible across +files; the shuffled run is what keeps that trade honest, because a test that passes only after a +fresh import fails there with a seed to replay. + +Document the replay command in the README of the tests, so that whoever reads the failing log knows +what to do with the number: + +```powershell +$configuration = New-PesterConfiguration +$configuration.Run.Path = Join-Path -Path $PSScriptRoot -ChildPath 'Tests/Unit' +$configuration.Run.Shuffle = $true +$configuration.Run.ShuffleSeed = 852659930 + +Invoke-Pester -Configuration $configuration +``` + +With the runner script the same replay is +`./Tests/Invoke-Tests.ps1 -TestType Unit -ShuffleSeed 852659930`. + +### Global Mocks + +With `Mock.Global` on, a mock is still removed when the block that declared it ends, and it is tied +to the run that created it, so it cannot leak into a nested run of Pester inside a test. Measured on +6.2.0 with a global mock of `Get-Date`: a test in a nested `Invoke-Pester` call got the real +command while the outer test still got the mock, and a `Describe` block after the one that declared +the mock got the real command again. + +The option may still change. The Pester team has stated an intent to make it the default in +version 7. diff --git a/.github/instructions/pester-supporting-docs/security-test-template.md b/.github/instructions/pester-supporting-docs/security-test-template.md index a1fbd7b..db16bff 100644 --- a/.github/instructions/pester-supporting-docs/security-test-template.md +++ b/.github/instructions/pester-supporting-docs/security-test-template.md @@ -1,11 +1,15 @@ # Security Test Template -A test file for functions that handle untrusted input, paths, or credentials. Write one only for a -function that has such a requirement (TST-43). The security rules that these tests check are in the -[security rules](../powershell-security.instructions.md), and the testing rules are in the -[Pester rules](../powershell-pester.instructions.md). The file below passes on Pester 6.2.0 under -PowerShell 7.6 and Windows PowerShell 5.1, and each of its tests fails against a copy of the module -that has the defect it looks for. +Use this template for validating security controls and input sanitization: test files for functions +that handle untrusted input, paths, credentials, access rights, or protected data. TST-43 requires +security tests for credential handling and input validation. The security rules that these tests +check are in the [security rules](../powershell-security.instructions.md), and the testing rules +are in the [Pester rules](../powershell-pester.instructions.md). The files below pass on Pester +6.2.0 under PowerShell 7.6 and Windows PowerShell 5.1, and each of their tests fails against a copy +of the code that has the defect it looks for. + +Do not use emoji in generated code, documentation, or test output. Use plain text and standard +ASCII characters only (STY-06, TST-07). ## The Code Under Test @@ -37,7 +41,18 @@ BeforeDiscovery { @{ ChildPath = '../outside.txt' } @{ ChildPath = 'reports/../../outside.txt' } @{ ChildPath = '../root-other/outside.txt' } + @{ ChildPath = '../../../../etc/shadow' } + ) + + # A backslash is a separator on Windows only, so these patterns are attacks there only. + $onWindows = $PSVersionTable.PSEdition -eq 'Desktop' -or $IsWindows + $windowsTraversalPatterns = @( + '..\outside.txt' + '..\..\Windows\System32\config' + '..\..\..\secrets\passwords.txt' + 'C:\Windows\System32\SAM' ) + $windowsTraversalCases = $windowsTraversalPatterns.ForEach({ @{ ChildPath = $_ } }) } BeforeAll { @@ -47,12 +62,12 @@ BeforeAll { } } -Describe 'Get-Thing input validation' -Tag 'Security' { - BeforeAll { +Describe 'Get-Thing input validation' -Tag 'Security', 'InputValidation' { + BeforeEach { Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { } } - It 'Rejects a name that contains ' -ForEach $injectionCases { + It 'Should reject a name that contains ' -ForEach $injectionCases { { Get-Thing -Name $Text } | Should-Throw -FullyQualifiedErrorId 'ParameterArgumentValidationError,Get-Thing' @@ -60,7 +75,7 @@ Describe 'Get-Thing input validation' -Tag 'Security' { } } -Describe 'Read-ThingFile path handling' -Tag 'Security' { +Describe 'Read-ThingFile path handling' -Tag 'Security', 'InputValidation' { BeforeAll { $root = Join-Path -Path $TestDrive -ChildPath 'root' $sibling = Join-Path -Path $TestDrive -ChildPath 'root-other' @@ -76,22 +91,27 @@ Describe 'Read-ThingFile path handling' -Tag 'Security' { Set-Content -LiteralPath $siblingPath -Value 'outside' -Encoding ascii } - It 'Reads a file inside the root' { + It 'Should read a file inside the root' { Read-ThingFile -Root $root -ChildPath 'inside.txt' | Should-BeString 'inside' -TrimWhitespace } - It 'Refuses , which resolves outside the root' -ForEach $traversalCases { + It 'Should refuse , which resolves outside the root' -ForEach $traversalCases { + { Read-ThingFile -Root $root -ChildPath $ChildPath } | + Should-Throw -FullyQualifiedErrorId 'PathOutsideRoot,Read-ThingFile' + } + + It 'Should refuse the Windows path ' -ForEach $windowsTraversalCases -Skip:(-not $onWindows) { { Read-ThingFile -Root $root -ChildPath $ChildPath } | Should-Throw -FullyQualifiedErrorId 'PathOutsideRoot,Read-ThingFile' } - It 'Refuses an absolute path outside the root' { + It 'Should refuse an absolute path outside the root' { { Read-ThingFile -Root $root -ChildPath $outsidePath } | Should-Throw -FullyQualifiedErrorId 'PathOutsideRoot,Read-ThingFile' } } -Describe 'Connect-ThingService credential handling' -Tag 'Security' { +Describe 'Connect-ThingService credential handling' -Tag 'Security', 'CredentialHandling' { BeforeAll { # The password is generated for each run, so the test file stores no secret. $secretText = [guid]::NewGuid().ToString('N') @@ -100,12 +120,12 @@ Describe 'Connect-ThingService credential handling' -Tag 'Security' { $secretPattern = [regex]::Escape($secretText) } - It 'Declares Credential as a PSCredential' { + It 'Should declare Credential as a PSCredential' { Get-Command -Name Connect-ThingService | Should-HaveParameter -ParameterName Credential -Type ([System.Management.Automation.PSCredential]) } - It 'Fails without prompting when no credential is passed' { + It 'Should fail without prompting when no credential is passed' { Mock -CommandName Get-Credential -ModuleName ModuleName -MockWith { } Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { } @@ -116,7 +136,31 @@ Describe 'Connect-ThingService credential handling' -Tag 'Security' { Should-NotInvoke -CommandName Invoke-RestMethod -ModuleName ModuleName } - It 'Does not write the password to any stream' { + It 'Should validate credential authentication' { + Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { + throw [System.UnauthorizedAccessException]::new('The service refused the credential.') + } + $validSignIn = @{ + CommandName = 'Invoke-RestMethod' + ModuleName = 'ModuleName' + ParameterFilter = { $Credential.UserName -eq 'svc-test' } + } + Mock @validSignIn -MockWith { [PSCustomObject]@{ sessionId = 'session-1' } } + $invalidCredential = [System.Management.Automation.PSCredential]::new('svc-unknown', $password) + $refused = @{ + FullyQualifiedErrorId = 'SignInFailed,Connect-ThingService' + ExceptionMessage = '*Sign-in*failed*' + } + + # The valid credential must work. With -ErrorAction Stop, any error fails the test. + $session = Connect-ThingService -Credential $credential -ErrorAction Stop + $session.SessionId | Should-Be 'session-1' + + # The invalid credential must fail. + { Connect-ThingService -Credential $invalidCredential } | Should-Throw @refused + } + + It 'Should not expose credentials in logs or output' { Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { [PSCustomObject]@{ sessionId = 'session-1' } } @@ -126,10 +170,11 @@ Describe 'Connect-ThingService credential handling' -Tag 'Security' { # The first assertion proves that the verbose stream was captured. $text | Should-MatchString 'Signing in' + $text | Should-NotMatchString ([regex]::Escape($credential.UserName)) $text | Should-NotMatchString $secretPattern } - It 'Does not put the password in the error when sign-in fails' { + It 'Should not expose the password in the error when sign-in fails' { Mock -CommandName Invoke-RestMethod -ModuleName ModuleName -MockWith { throw [System.UnauthorizedAccessException]::new('The service refused the credential.') } @@ -143,6 +188,467 @@ Describe 'Connect-ThingService credential handling' -Tag 'Security' { } ``` +## Tests Against Stand-In Functions + +The example module has no e-mail parameter, credential cache, role check, encryption, or password +hashing. The tests for those controls therefore run against a small stand-in module that the +template defines itself. In a project, the functions under test are the project's own. +Comment-based help is omitted here to keep the example short. + +```powershell +# Tests/TestHelpers/SecuritySample.psm1 +Set-StrictMode -Version Latest + +$script:credentialCache = @{} + + +function ConvertTo-ThingOwner { + [CmdletBinding()] + [OutputType('SecuritySample.Owner')] + param( + [Parameter(Mandatory)] + [ValidateLength(3, 254)] + [ValidatePattern('^[^@\s.]+(\.[^@\s.]+)*@[^@\s.]+(\.[^@\s.]+)+$')] + [string]$EmailAddress, + + [Parameter(Mandatory)] + [ValidateLength(1, 256)] + [ValidateScript({ -not [string]::IsNullOrWhiteSpace($_) })] + [string]$DisplayName + ) + + [PSCustomObject]@{ + PSTypeName = 'SecuritySample.Owner' + EmailAddress = $EmailAddress + DisplayName = $DisplayName + } +} + + +function Add-CachedCredential { + [CmdletBinding(SupportsShouldProcess)] + param( + [Parameter(Mandatory)] + [System.Management.Automation.PSCredential] + [System.Management.Automation.Credential()] + $Credential + ) + + # The cache keeps the PSCredential object, so the password stays a SecureString. + if ($PSCmdlet.ShouldProcess($Credential.UserName, 'Cache credential')) { + $script:credentialCache[$Credential.UserName] = $Credential + } +} + + +function Get-CredentialCache { + [CmdletBinding()] + [OutputType('SecuritySample.CacheEntry')] + param() + + foreach ($cachedCredential in $script:credentialCache.Values) { + [PSCustomObject]@{ + PSTypeName = 'SecuritySample.CacheEntry' + Credential = $cachedCredential + } + } +} + + +function Get-CurrentUserRole { + [CmdletBinding()] + [OutputType([string])] + param() + + 'User' +} + + +function Get-CurrentSession { + [CmdletBinding()] + [OutputType('SecuritySample.Session')] + param() + + # There is no session until a test supplies one. +} + + +function Test-PathAccess { + [CmdletBinding()] + [OutputType([bool])] + param( + [Parameter(Mandatory)] + [string]$Path + ) + + Test-Path -LiteralPath $Path +} + + +function Invoke-PrivilegedAction { + [CmdletBinding()] + param( + [string]$Path = '' + ) + + Write-Verbose "The privileged action ran for '$Path'." +} + + +function Invoke-AdminTask { + [CmdletBinding()] + param( + [ValidateSet('User', 'Administrator')] + [string]$RequiredRole = 'Administrator', + + [string]$Path = '', + + [switch]$RequireValidSession + ) + + $denial = $null + if ($RequireValidSession) { + $session = Get-CurrentSession + if ($null -eq $session -or -not $session.Valid) { + $denial = 'InvalidSession', 'Access denied: invalid session.' + } + } + + if ($null -eq $denial -and (Get-CurrentUserRole) -ne $RequiredRole) { + $denial = 'InsufficientPrivileges', 'Access denied: insufficient privileges.' + } + + if ($null -eq $denial -and $Path -and -not (Test-PathAccess -Path $Path)) { + $denial = 'PathAccessDenied', 'Access denied to the path.' + } + + if ($null -ne $denial) { + $exception = [System.UnauthorizedAccessException]::new($denial[1]) + $category = [System.Management.Automation.ErrorCategory]::PermissionDenied + $errorRecord = [System.Management.Automation.ErrorRecord]::new($exception, $denial[0], $category, $Path) + $PSCmdlet.ThrowTerminatingError($errorRecord) + } + + Invoke-PrivilegedAction -Path $Path +} + + +function Protect-ThingData { + [CmdletBinding()] + [OutputType('SecuritySample.ProtectedData')] + param( + [Parameter(Mandatory)] + [string]$Data, + + [Parameter(Mandatory)] + [ValidateCount(32, 32)] + [byte[]]$Key + ) + + $aes = [System.Security.Cryptography.Aes]::Create() + try { + $aes.Key = $Key + $aes.GenerateIV() + $plainBytes = [System.Text.Encoding]::UTF8.GetBytes($Data) + $cipherBytes = $aes.CreateEncryptor().TransformFinalBlock($plainBytes, 0, $plainBytes.Length) + $initializationVector = $aes.IV + } catch { + $PSCmdlet.ThrowTerminatingError($_) + } finally { + $aes.Dispose() + } + + [PSCustomObject]@{ + PSTypeName = 'SecuritySample.ProtectedData' + EncryptedData = [System.Convert]::ToBase64String($cipherBytes) + EncryptionAlgorithm = 'AES-256-CBC' + IV = [System.Convert]::ToBase64String($initializationVector) + } +} + + +function Invoke-SecureTransmission { + [CmdletBinding()] + param( + [Parameter(Mandatory)] + [string]$Payload, + + [Parameter(Mandatory)] + [uri]$Endpoint + ) + + try { + $null = Invoke-RestMethod -Uri $Endpoint -Method Post -Body $Payload -ErrorAction Stop + } catch { + $PSCmdlet.ThrowTerminatingError($_) + } +} + + +function Send-ThingData { + [CmdletBinding()] + [OutputType('SecuritySample.TransmissionResult')] + param( + [Parameter(Mandatory)] + [string]$Data, + + [Parameter(Mandatory)] + [ValidateCount(32, 32)] + [byte[]]$Key, + + [Parameter(Mandatory)] + [uri]$Endpoint + ) + + $protectedData = Protect-ThingData -Data $Data -Key $Key + Invoke-SecureTransmission -Payload $protectedData.EncryptedData -Endpoint $Endpoint + + [PSCustomObject]@{ + PSTypeName = 'SecuritySample.TransmissionResult' + Success = $true + Encrypted = $true + } +} + + +function ConvertTo-PasswordHash { + [CmdletBinding()] + [OutputType('SecuritySample.PasswordHash')] + param( + [Parameter(Mandatory)] + [System.Security.SecureString]$Password + ) + + $salt = [byte[]]::new(16) + $generator = [System.Security.Cryptography.RandomNumberGenerator]::Create() + try { + $generator.GetBytes($salt) + $plainText = [System.Net.NetworkCredential]::new('', $Password).Password + $deriveBytes = [System.Security.Cryptography.Rfc2898DeriveBytes]::new( + $plainText, $salt, 100000, [System.Security.Cryptography.HashAlgorithmName]::SHA256) + $hash = $deriveBytes.GetBytes(32) + $deriveBytes.Dispose() + } catch { + $PSCmdlet.ThrowTerminatingError($_) + } finally { + $generator.Dispose() + } + + [PSCustomObject]@{ + PSTypeName = 'SecuritySample.PasswordHash' + HashedPassword = [System.Convert]::ToBase64String($hash) + Salt = [System.Convert]::ToBase64String($salt) + Algorithm = 'PBKDF2' + } +} +``` + +The test file for these functions: + +```powershell +#Requires -Modules @{ ModuleName = 'Pester'; ModuleVersion = '6.2.0' } + +BeforeDiscovery { + $invalidEmailPatterns = @( + 'notanemail' + '@domain.com' + 'user@' + 'user..name@domain.com' + 'user@domain' + ) + $invalidEmailCases = $invalidEmailPatterns.ForEach({ @{ InvalidEmail = $_ } }) +} + +BeforeAll { + $helperPath = Join-Path -Path $PSScriptRoot -ChildPath 'TestHelpers' + if (-not (Get-Module -Name SecuritySample)) { + Import-Module -Name (Join-Path -Path $helperPath -ChildPath 'SecuritySample.psm1') + } + + # The password is generated for each run, so the test file stores no secret. + $secretText = [guid]::NewGuid().ToString('N') + $password = ConvertTo-SecureString -String $secretText -AsPlainText -Force + $credential = [System.Management.Automation.PSCredential]::new('cache-test', $password) + $secretPattern = [regex]::Escape($secretText) +} + +Describe 'ConvertTo-ThingOwner input validation' -Tag 'Security', 'InputValidation' { + BeforeAll { + $rejectedAddress = @{ + FullyQualifiedErrorId = 'ParameterArgumentValidationError,ConvertTo-ThingOwner' + ExceptionMessage = '*EmailAddress*' + } + $rejectedName = @{ + FullyQualifiedErrorId = 'ParameterArgumentValidationError*,ConvertTo-ThingOwner' + ExceptionMessage = '*DisplayName*' + } + } + + It 'Should accept a valid address and name' { + $owner = ConvertTo-ThingOwner -EmailAddress 'ann@example.com' -DisplayName 'Ann' + + $owner.EmailAddress | Should-Be 'ann@example.com' + } + + It 'Should reject invalid email format: ' -ForEach $invalidEmailCases { + { ConvertTo-ThingOwner -EmailAddress $InvalidEmail -DisplayName 'Ann' } | + Should-Throw @rejectedAddress + } + + It 'Should validate parameter length limits' { + $oversizedInput = 'x' * 10000 + + { ConvertTo-ThingOwner -EmailAddress 'ann@example.com' -DisplayName $oversizedInput } | + Should-Throw @rejectedName + } + + It 'Should reject input' -ForEach @( + @{ Description = 'null'; Value = $null } + @{ Description = 'empty'; Value = '' } + @{ Description = 'whitespace-only'; Value = ' ' } + ) { + { ConvertTo-ThingOwner -EmailAddress 'ann@example.com' -DisplayName $Value } | + Should-Throw @rejectedName + } +} + +Describe 'Credential caching' -Tag 'Security', 'CredentialHandling' { + It 'Should implement secure credential caching' { + Add-CachedCredential -Credential $credential + + $cacheContent = @(Get-CredentialCache) + if ($cacheContent.Count -eq 0) { + Set-ItResult -Inconclusive -Because 'no credential cache was produced to inspect' + } + + $cacheText = $cacheContent | Format-List -Property * | Out-String + $cacheText | Should-MatchString 'Credential' + $cacheText | Should-NotMatchString $secretPattern + $cacheText | Should-NotMatchString ([regex]::Escape($credential.UserName)) + } +} + +Describe 'Invoke-AdminTask access control' -Tag 'Security', 'AccessControl' { + BeforeEach { + Mock -CommandName Get-CurrentUserRole -ModuleName SecuritySample -MockWith { 'Administrator' } + Mock -CommandName Invoke-PrivilegedAction -ModuleName SecuritySample -MockWith { } + } + + It 'Should deny access when the role is insufficient' { + Mock -CommandName Get-CurrentUserRole -ModuleName SecuritySample -MockWith { 'User' } + $denied = @{ + FullyQualifiedErrorId = 'InsufficientPrivileges,Invoke-AdminTask' + ExceptionMessage = '*insufficient privileges*' + } + + { Invoke-AdminTask -RequiredRole 'Administrator' } | Should-Throw @denied + + # The assertion that matters: the action never ran. + Should-NotInvoke -CommandName Invoke-PrivilegedAction -ModuleName SecuritySample + } + + It 'Should allow access when the role is sufficient' { + Invoke-AdminTask -RequiredRole 'Administrator' -ErrorAction Stop + + Should-Invoke -CommandName Invoke-PrivilegedAction -ModuleName SecuritySample -Times 1 -Exactly + } + + It 'Should validate file system permissions' { + Mock -CommandName Test-PathAccess -ModuleName SecuritySample -MockWith { $false } + $restrictedPath = Join-Path -Path $TestDrive -ChildPath 'restricted' + $denied = @{ + FullyQualifiedErrorId = 'PathAccessDenied,Invoke-AdminTask' + ExceptionMessage = '*access denied*' + } + + { Invoke-AdminTask -Path $restrictedPath } | Should-Throw @denied + + Should-NotInvoke -CommandName Invoke-PrivilegedAction -ModuleName SecuritySample + } + + It 'Should reject an invalid session' { + Mock -CommandName Get-CurrentSession -ModuleName SecuritySample -MockWith { } + $denied = @{ + FullyQualifiedErrorId = 'InvalidSession,Invoke-AdminTask' + ExceptionMessage = '*invalid session*' + } + + { Invoke-AdminTask -RequireValidSession } | Should-Throw @denied + + Should-NotInvoke -CommandName Invoke-PrivilegedAction -ModuleName SecuritySample + } + + It 'Should accept a valid session' { + Mock -CommandName Get-CurrentSession -ModuleName SecuritySample -MockWith { + [PSCustomObject]@{ Valid = $true; UserId = 'test-user' } + } + + Invoke-AdminTask -RequireValidSession -ErrorAction Stop + + Should-Invoke -CommandName Invoke-PrivilegedAction -ModuleName SecuritySample -Times 1 -Exactly + } +} + +Describe 'Data protection' -Tag 'Security', 'DataProtection' { + BeforeAll { + # The key is generated for each run as well. + $key = [byte[]]::new(32) + $generator = [System.Security.Cryptography.RandomNumberGenerator]::Create() + $generator.GetBytes($key) + $generator.Dispose() + } + + It 'Should encrypt sensitive data at rest' { + $sensitiveData = 'Confidential Information 12345' + + $result = Protect-ThingData -Data $sensitiveData -Key $key + + # The data is encrypted: it is not plain text, and not plain text in another encoding. + $result.EncryptedData | Should-NotBeEmptyString + $result.EncryptedData | Should-NotMatchString 'Confidential Information' + $storedBytes = [System.Convert]::FromBase64String($result.EncryptedData) + [System.Text.Encoding]::UTF8.GetString($storedBytes) | Should-NotMatchString 'Confidential Information' + + # The encryption metadata is present. + $result.EncryptionAlgorithm | Should-Be 'AES-256-CBC' + $result.IV | Should-NotBeEmptyString + } + + It 'Should implement secure data transmission' { + Mock -CommandName Invoke-SecureTransmission -ModuleName SecuritySample -MockWith { } + $serviceEndpoint = 'https://api.example.test/' + + $result = Send-ThingData -Data 'Sensitive transmission data' -Key $key -Endpoint $serviceEndpoint + + $result.Success | Should-BeTrue + $result.Encrypted | Should-BeTrue + $encryptedTransmission = @{ + CommandName = 'Invoke-SecureTransmission' + ModuleName = 'SecuritySample' + ParameterFilter = { $Payload -and $Payload -notmatch 'Sensitive transmission data' } + } + Should-Invoke @encryptedTransmission -Times 1 -Exactly + } + + It 'Should securely hash passwords' { + $result = ConvertTo-PasswordHash -Password $password + $second = ConvertTo-PasswordHash -Password $password + + # The password is hashed, not stored as plain text. + $result.HashedPassword | Should-NotBeEmptyString + $result.HashedPassword | Should-NotMatchString $secretPattern + + # A salt is used, so the same password gives a different hash each time. + $result.Salt | Should-NotBeEmptyString + $second.HashedPassword | Should-NotBe $result.HashedPassword + + # The algorithm is an accepted one. There is no Should-BeIn; Should-Any over the list + # states the same thing. + 'SHA256', 'SHA512', 'bcrypt', 'scrypt', 'PBKDF2' | Should-Any { $_ -eq $result.Algorithm } + } +} +``` + ## What The Template Shows ### Attack Data Is Built During Discovery @@ -157,8 +663,11 @@ cases are derived from it: | `foreach ($item in $list) { @{ Text = $item } }` | No cases. Discovery of the file fails. | | `$list.ForEach({ @{ Text = $_ } })` | No cases. Discovery of the file fails. | -Do not build cases with `ForEach-Object`. Do not add `-AllowNullOrEmptyForEach` in a security test -file either: an empty list of attacks must stop the run. +Turn a list of patterns into cases with the `.ForEach()` method, as the Windows paths and the +e-mail addresses do. Write the hashtables directly, as the injection cases do, when a case needs a +description beside its value. Do not build cases with `ForEach-Object`. Do not add +`-AllowNullOrEmptyForEach` in a security test file either: an empty list of attacks must stop the +run. To see how many tests a file produces without running them: @@ -184,8 +693,14 @@ the test runs, and the function receives the output of the command instead of th @{ Description = 'a subexpression'; Text = "$(Get-Process)" } ``` -Write path separators as `/`. A backslash is a separator on Windows only, so `..\outside.txt` names -an ordinary file inside the root on Linux and macOS. +### Both Kinds Of Path Separator Are Tested + +A pattern written with `/` is an attack on every platform. A pattern written with `\`, and a +Windows absolute path such as `C:\Windows\System32\SAM`, is an attack on Windows only: a backslash +is a separator there and an ordinary character on Linux and macOS, where `..\outside.txt` names a +file inside the root. The template therefore keeps the two lists apart and skips the Windows list +on other platforms. A function that is tested with one kind of separator only can still be open to +the other. ### The Unsafe Outcome Must Be Possible @@ -197,18 +712,23 @@ A test proves a control only if it fails when the control is missing: the service. An error ID alone would not show that. - The first assertion on the captured text shows that the verbose stream is in it. An empty capture would pass the second assertion without proving anything. +- A valid address and name are accepted, so the rejections are not the work of a function that + rejects everything. +- The encryption test also decodes the stored value. A function that only encoded the text as + Base64 passed the test without that assertion. ### No Secret Is Stored In The Test File SEC-01 and SEC-04 apply to test files. The password is a value generated at run time, converted with `ConvertTo-SecureString -AsPlainText` only because it is not a literal and is not stored in source. A random value is also a better marker than a fixed one: it cannot appear in the output by -coincidence. +coincidence. The encryption key is generated the same way. ### Every Stream Is Captured `*>&1` merges the error, warning, verbose, debug, and information streams into the output. The call -passes `-Verbose` and `-InformationAction Continue` so that those streams are written at all. +passes `-Verbose` and `-InformationAction Continue` so that those streams are written at all. The +test then asserts that neither the user name nor the password is in the captured text. Do not add `-Debug` in a test that must run on Windows PowerShell 5.1. There it makes `Write-Debug` ask for confirmation, which fails in a non-interactive session. On PowerShell 7 `-Debug` only turns @@ -217,9 +737,14 @@ the stream on. ### A Value Used As A Pattern Is Escaped `Should-MatchString` and `Should-NotMatchString` take a regular expression. A password or a -connection string can contain characters such as `.`, `$`, and `(`, so pass it through -`[regex]::Escape()`. An invalid pattern stops the test with an error, and a valid but unintended one -can fail to match the text it was meant to find. +connection string can contain characters such as `.`, `$`, `(`, and `\`, so pass it through +`[regex]::Escape()`. Measured on Pester 6.2.0 with `Should-NotMatchString`: + +| Unescaped value | What happens | +| --- | --- | +| `Pa$$w0rd`, in a text that contains it | The assertion passes. `$` anchors the pattern, so the leak is not found. | +| `db.example`, in a text that contains `dbXexample` | The assertion fails. `.` matches any character, so other text is reported as the secret. | +| `abc(def` | The test stops with an error about an invalid pattern. | ### The Error Is Asserted, Not Only Caught @@ -227,9 +752,76 @@ can fail to match the text it was meant to find. assertions in the `catch` block of a `try` statement: when the function does not throw, the block does not run and the test passes. +A rejection is asserted by its error ID. Where the message is part of what the caller sees, the +same call adds a message pattern, which needs wildcards: `-ExceptionMessage '*access denied*'`. +For a parameter that a validation attribute rejects, the pattern names the parameter, because +several parameters share the error ID `ParameterArgumentValidationError`. + +### Both Outcomes Of A Sign-In + +The sign-in test proves that a valid credential works and that an invalid one fails. There is no +`Should-NotThrow`: the valid call is made directly with `-ErrorAction Stop`, and any error fails +the test (TST-16). + ### What The Function Must Not Do For a refusal, `Should-NotInvoke` on the protected action is stronger evidence than `Should-Throw`, -because a function can throw after it has done the damage. For SEC-05, mock `Get-Credential` and -`Read-Host` in the module and assert that neither was called; the mock also keeps a defective -function from waiting for input. +because a function can throw after it has done the damage. The access-control tests assert both for +a role, a path, and a session. For SEC-05, mock `Get-Credential` and `Read-Host` in the module and +assert that neither was called; the mock also keeps a defective function from waiting for input. + +### The Shape Of An Object + +There is no `Should -HaveProperty` operator in Pester, and no `Should-HaveProperty` assertion. +Assert the shape instead: the count, and then the properties of every item. + +```powershell +$records | Should-BeCollection -Count 5 +$records | Should-All { $null -ne $_.UserId -and $null -ne $_.Action -and $null -ne $_.Timestamp } +``` + +### A Test That Has Nothing To Inspect + +The caching test calls `Set-ItResult -Inconclusive` when no cache was produced. The test is then +reported as inconclusive, not as passed, so a missing cache cannot look like a safe one. + +## Security Test Guidelines + +### Input Validation Testing + +- Test all forms of malicious input. +- Validate parameter length limits. +- Test encoding and special characters. +- Verify path traversal prevention. +- Test command injection attempts. + +### Credential Security Testing + +- Verify that credentials are never exposed in logs or output. +- Test `SecureString` handling. +- Validate authentication mechanisms. +- Test the security of credential caching. +- Verify secure transmission of credentials. + +### Access Control Testing + +- Test role-based access control. +- Validate file system permissions. +- Test session management. +- Verify that authorization is enforced. +- Test the prevention of privilege escalation. + +### Data Protection Testing + +- Test encryption at rest and in transit. +- Validate secure data handling. +- Test password hashing algorithms. +- Verify secure deletion methods. +- Test data anonymization features. + +### Error Handling Security + +- Ensure that error messages hold no sensitive data. +- Test what information an error discloses. +- Test error handling under attack conditions. +- Verify graceful failure modes. diff --git a/.github/instructions/pester-supporting-docs/test-data-guide.md b/.github/instructions/pester-supporting-docs/test-data-guide.md index fab7817..04d2ef6 100644 --- a/.github/instructions/pester-supporting-docs/test-data-guide.md +++ b/.github/instructions/pester-supporting-docs/test-data-guide.md @@ -5,6 +5,9 @@ creates, and helper functions that build objects. It supports rules TST-02, TST- and TST-31 in [powershell-pester.instructions.md](../powershell-pester.instructions.md). The behaviour described here was checked on Pester 6.2.0. +Do not use emoji in generated code, documentation, or test output. Use plain text and standard +ASCII characters only (STY-06, TST-07). + ## Discovery-Time and Run-Time Data Pester reads a test file twice. During discovery it runs the file to find the `Describe`, `Context`, @@ -61,8 +64,9 @@ A file that fails discovery adds nothing to `FailedCount`. It sets `FailedContai the run result `Failed`, which is why the CI gate tests `Result` (TST-42). The check for an empty list is the `Run.FailOnNullOrEmptyForEach` option, which is on by default. -`-AllowNullOrEmptyForEach` on a `Describe`, `Context`, or `It` turns it off for that block. TST-30 -requires a list that is not empty, so correct the data instead of adding the switch. +`-AllowNullOrEmptyForEach` on a `Describe`, `Context`, or `It` turns it off for that block. Opt out +with the switch where an empty list is legitimate, or correct the code that builds the list so that +it cannot be empty. In every other case TST-30 requires a list that is not empty. ### Each File Builds Its Own Data @@ -135,13 +139,65 @@ It 'Converts to ' -ForEach @( ## Deterministic Data -Tests MUST NOT depend on the current time or on values that change between runs (TST-31). - -- Cases for `-ForEach` are fixed values. Do not build them with `Get-Random`, `Get-Date`, or - `[guid]::NewGuid()`: the test names would change on every run, and a failure could not be matched - with an earlier one. +Tests MUST NOT depend on the current time (TST-31), and the data that feeds `-ForEach` must be +stable. + +- For cases, prefer fixed fixtures, such as a `.psd1` file loaded with + `Import-PowerShellDataFile`, or a seeded generator. Do not build cases with an unseeded + `Get-Random`, with `Get-Date`, or with `[guid]::NewGuid()`: the test names would change on every + run and in every runspace, and a failure could not be matched with an earlier one or with a + report. +- Keep `Get-Random` for values that are used inside `It` bodies, where the name is already fixed. + The assertion must then hold for every value that can come out, and the failure message should + state the value. - When the code under test reads the clock, mock `Get-Date`. -- When the code under test needs a unique value, pass a fixed one from the test. +- When the code under test needs a unique value and the test asserts on it, pass a fixed one from + the test. + +A seeded generator returns the same values on every run. With the seed below, the three cases were +563, 218, and 838 on PowerShell 7.6 and on Windows PowerShell 5.1. `Get-ThingSummary` is the +function of the [performance test template](./performance-test-template.md); it emits one object +for each state, with the number of things in that state. + +```powershell +BeforeDiscovery { + # Stable names: the same test identity on every run and in every runspace. + $random = [System.Random]::new(20260331) + $loadCases = foreach ($number in 1..3) { + @{ ItemCount = $random.Next(100, 1000) } + } +} + +Describe 'Get-ThingSummary' -Tag 'Unit', 'Public' { + It 'Counts things' -ForEach $loadCases { + $things = foreach ($index in 1..$ItemCount) { + [PSCustomObject]@{ Name = "thing-$index"; State = 'ready' } + } + + $summary = @(Get-ThingSummary -Thing $things) + + $summary | Should-BeCollection -Count 1 + $summary[0].Count | Should-Be $ItemCount + } + + It 'Counts a number of things that is chosen at run time' { + $thingCount = Get-Random -Minimum 1 -Maximum 100 + $things = foreach ($index in 1..$thingCount) { + [PSCustomObject]@{ Name = "thing-$index"; State = 'ready' } + } + + $summary = @(Get-ThingSummary -Thing $things) + + $summary | Should-BeCollection -Count 1 + $summary[0].Count | Should-Be $thingCount -Because "the input held $thingCount things" + } +} +``` + +`Get-Random -SetSeed` also repeats its values, but it reseeds the generator of the whole session; +an object of `[System.Random]` keeps the seed to the code that owns it. + +When the code under test reads the clock, the test mocks `Get-Date`: ```powershell Describe 'Test-ReportStale' -Tag 'Unit' { @@ -162,18 +218,47 @@ Describe 'Test-ReportStale' -Tag 'Unit' { ## Data Files -Keep fixed data beside the tests: +Keep fixed data beside the tests, sorted by what it is for: ```text Tests/ |-- TestData/ -| |-- server-cases.psd1 cases for -ForEach -| |-- users.json input that the code under test reads -| `-- api-responses.json responses that mocks return +| |-- server-cases.psd1 cases for -ForEach +| |-- Configurations/ settings for each environment +| | |-- development.psd1 +| | |-- testing.psd1 +| | `-- production.psd1 +| |-- SampleData/ input that the code under test reads +| | |-- users.json +| | |-- servers.csv +| | `-- configurations.xml +| |-- MockResponses/ responses that mocks return +| | |-- api-responses.json +| | |-- database-results.json +| | `-- service-responses.json +| |-- Templates/ +| | |-- user-template.json +| | |-- server-template.json +| | `-- config-template.psd1 +| |-- Baselines/ figures that tests compare with +| | |-- performance-baselines.psd1 +| | `-- security-baselines.json +| `-- Fixtures/ +| |-- test-certificates/ +| |-- test-files/ +| `-- sample-logs/ `-- TestHelpers/ - `-- TestData.ps1 functions that build test objects + |-- TestDataFactory.ps1 builds test objects + |-- MockDataGenerator.ps1 + `-- DataValidation.ps1 checks a data set ``` +A suite with a handful of data files can keep them directly in `TestData`, with one helper file +such as `TestHelpers/TestData.ps1`, and add the subfolders when the number of files grows. + +The samples in this guide are written for a test file one level below `Tests`. For a file in +`Tests/Unit/Public`, add one more `Split-Path` to reach the `Tests` folder. + A `.psd1` file suits cases, because `Import-PowerShellDataFile` returns hashtables without running code: @@ -219,7 +304,8 @@ A JSON file suits input that the code under test parses, and responses that a mo BeforeAll { $testsRoot = Split-Path -Path $PSScriptRoot -Parent $dataPath = Join-Path -Path $testsRoot -ChildPath 'TestData' - $userJson = Get-Content -LiteralPath (Join-Path -Path $dataPath -ChildPath 'users.json') -Raw + $samplePath = Join-Path -Path $dataPath -ChildPath 'SampleData' + $userJson = Get-Content -LiteralPath (Join-Path -Path $samplePath -ChildPath 'users.json') -Raw $users = ConvertFrom-Json -InputObject $userJson } ``` @@ -282,11 +368,194 @@ Describe 'Get-DeploymentRegion' -Tag 'Unit' { } ``` +## Test Data Factory + +When many tests need the same kinds of object, build them in one place: a `TestDataFactory` class +with one method for each kind of object and a cache, and a wrapper function for each method. The +generator is seeded, so the factory returns the same objects on every run. Comment-based help is +omitted here to keep the example short. + +```powershell +# Tests/TestHelpers/TestDataFactory.ps1 + +class TestDataFactory { + static [hashtable] $Cache = @{} + + static [object[]] CreateUsers([int]$Count, [string]$Type) { + $cacheKey = "Users_${Count}_$Type" + if ([TestDataFactory]::Cache.ContainsKey($cacheKey)) { + return [TestDataFactory]::Cache[$cacheKey] + } + + # The seed makes the generator return the same users on every run. + $random = [System.Random]::new(20260331) + $firstNames = 'James', 'Mary', 'John', 'Patricia', 'Robert', 'Jennifer' + $lastNames = 'Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Garcia' + $departments = 'IT', 'HR', 'Finance', 'Marketing', 'Operations' + $users = foreach ($id in 1..$Count) { + $firstName = $firstNames[$random.Next($firstNames.Count)] + $lastName = $lastNames[$random.Next($lastNames.Count)] + [PSCustomObject]@{ + PSTypeName = 'Test.User' + Id = $id + FullName = "$firstName $lastName" + Email = "$firstName.$lastName@example.com".ToLowerInvariant() + Department = $departments[$random.Next($departments.Count)] + Active = $Type -ne 'Disabled' + IsAdmin = $Type -eq 'Admin' + } + } + + [TestDataFactory]::Cache[$cacheKey] = $users + return $users + } + + static [void] ClearCache() { + [TestDataFactory]::Cache = @{} + } + + static [void] SaveTestData([string]$Name, [object]$Data, [string]$Folder) { + $filePath = Join-Path -Path $Folder -ChildPath "$Name.json" + $json = ConvertTo-Json -InputObject $Data -Depth 10 + Set-Content -LiteralPath $filePath -Value $json -Encoding ascii + } +} + + +function New-TestUserList { + [Diagnostics.CodeAnalysis.SuppressMessageAttribute( + 'PSUseShouldProcessForStateChangingFunctions', '', + Justification = 'Creates objects in memory and changes no state.')] + [CmdletBinding()] + [OutputType('Test.User')] + param( + [ValidateRange(1, 100000)] + [int]$Count = 5, + + [ValidateSet('Standard', 'Admin', 'Disabled')] + [string]$Type = 'Standard' + ) + + [TestDataFactory]::CreateUsers($Count, $Type) +} + + +function Clear-TestDataCache { + [CmdletBinding()] + param() + + [TestDataFactory]::ClearCache() +} +``` + +Add a method and a wrapper function for each further kind of object that the suite needs, such as +servers, configurations, API responses, and log entries. `SaveTestData` writes a data set to a file +for a test that needs one; pass it a folder under `$TestDrive` (TST-03). The attribute on +`New-TestUserList` is explained under +[Functions That Build Test Objects](#functions-that-build-test-objects). + +A test file dot-sources the factory in its top-level `BeforeAll`, and in `BeforeDiscovery` as well +when `-ForEach` data comes from it: + +```powershell +BeforeDiscovery { + $testsRoot = Split-Path -Path $PSScriptRoot -Parent + $helperPath = Join-Path -Path $testsRoot -ChildPath 'TestHelpers' + . (Join-Path -Path $helperPath -ChildPath 'TestDataFactory.ps1') + + # Shapes the test tree: one It for each user. + $userCases = (New-TestUserList -Count 3).ForEach({ @{ FullName = $_.FullName; User = $_ } }) +} + +BeforeAll { + $testsRoot = Split-Path -Path $PSScriptRoot -Parent + $helperPath = Join-Path -Path $testsRoot -ChildPath 'TestHelpers' + . (Join-Path -Path $helperPath -ChildPath 'TestDataFactory.ps1') +} + +Describe 'Test users' -Tag 'Unit', 'Public' { + It 'Gives an address at the example domain' -ForEach $userCases { + $User.Email | Should-MatchString '@example\.com$' + } +} +``` + +With the seed above, the three names are `Patricia Smith`, `Jennifer Jones`, and `Jennifer Brown` +on PowerShell 7.6 and on Windows PowerShell 5.1. + +### Data Isolation + +Create isolated test data for each test, and clean it up after the test: + +```powershell +Describe 'Factory users' -Tag 'Unit', 'Public' { + BeforeEach { + # Create isolated test data for each test. + $script:testUsers = New-TestUserList -Count 3 -Type 'Standard' + } + + AfterEach { + # Clean up the test data. + Clear-TestDataCache + Remove-Variable -Name 'testUsers' -Scope Script -ErrorAction SilentlyContinue + } + + It 'Builds three active users' { + $script:testUsers | Should-BeCollection -Count 3 + $script:testUsers | Should-All { $_.Active -and -not $_.IsAdmin } + } +} +``` + +The factory returns the cached objects for a request that it has seen, so a test that changes one +of them would change it for the next test. Clearing the cache in `AfterEach` gives the next test +objects of its own. + +### Performance Test Data + +Generate large data sets for performance tests with a shared helper in the same file: + +```powershell +function New-PerformanceTestData { + [Diagnostics.CodeAnalysis.SuppressMessageAttribute( + 'PSUseShouldProcessForStateChangingFunctions', '', + Justification = 'Creates objects in memory and changes no state.')] + [CmdletBinding()] + [OutputType('Test.User', 'Test.LargeText')] + param( + [ValidateRange(1, 100000)] + [int]$ItemCount = 1000, + + [ValidateSet('Users', 'LargeText')] + [string]$DataType = 'Users' + ) + + # Generate large data sets for performance tests. + switch ($DataType) { + 'Users' { + [TestDataFactory]::CreateUsers($ItemCount, 'Standard') + } + 'LargeText' { + # About 2.7 KB of text for each item. + $content = 'Lorem ipsum dolor sit amet ' * 100 + foreach ($id in 1..$ItemCount) { + [PSCustomObject]@{ + PSTypeName = 'Test.LargeText' + Id = $id + Content = $content + } + } + } + } +} +``` + ## Functions That Build Test Objects -When many tests need the same kind of object, write a small function that builds it from fixed -defaults, and let each test pass only the values it cares about. Keep the function free of random -values and of the clock. Comment-based help is omitted here to keep the example short. +A suite that needs one kind of object can do without the factory. Write a small function that +builds the object from fixed defaults, and let each test pass only the values it cares about. Keep +the function free of random values and of the clock. Comment-based help is omitted here to keep the +example short. ```powershell # Tests/TestHelpers/TestData.ps1 @@ -357,6 +626,29 @@ BeforeAll { } ``` +When several test files need a credential, the same code is a helper function beside the factory: + +```powershell +function New-TestCredential { + [Diagnostics.CodeAnalysis.SuppressMessageAttribute( + 'PSUseShouldProcessForStateChangingFunctions', '', + Justification = 'Creates an object in memory and changes no state.')] + [CmdletBinding()] + [OutputType([System.Management.Automation.PSCredential])] + param( + [string]$UserName = 'test-user' + ) + + # The password is created at run time, so no secret is stored in source. + $password = [System.Security.SecureString]::new() + foreach ($character in [guid]::NewGuid().ToString('N').ToCharArray()) { + $password.AppendChar($character) + } + + [System.Management.Automation.PSCredential]::new($UserName, $password) +} +``` + The tests assert on the user name or on the call that received the credential, never on the password. A test that needs a real secret is an integration test, and the secret comes from the CI system (CIC-06). @@ -376,7 +668,14 @@ It 'Loads three users that all have a name and a known department' { `Should-All` fails for an empty collection and names the items that did not pass the filter. A `foreach` loop with an assertion inside it passes when the collection is empty, because the -assertion never runs. +assertion never runs. When the collection has items, the loop stops at the first failure and does +not say which item it was. Measured on Pester 6.2.0 with the items 1, 2, 5, 4, 7 and the limit 5: +the loop reported `Actual: [int] 5` and nothing about 7, and `Should-All` reported +`2 of them @(5, 7) did not pass the filter`. + +There is no `Should-BeIn` assertion. Test membership with `-in` inside the filter, as the last line +of the example does, or with `Should-Any` over the allowed values. For a single value, the +[assertion guide](./assertion-guide.md) pipes the list to `Should-ContainCollection`. ## Data in Parallel Runs diff --git a/.github/instructions/pester-supporting-docs/test-execution.md b/.github/instructions/pester-supporting-docs/test-execution.md index 5960ab1..92c5f0c 100644 --- a/.github/instructions/pester-supporting-docs/test-execution.md +++ b/.github/instructions/pester-supporting-docs/test-execution.md @@ -5,6 +5,9 @@ parallel and shuffled. It supports rules TST-40 to TST-42 in [powershell-pester.instructions.md](../powershell-pester.instructions.md). The behaviour described here was checked on Pester 6.2.0 under PowerShell 7.6 and Windows PowerShell 5.1. +Do not use emoji in generated code, in documentation, or in test output. Use plain text and ASCII +characters only. + ## Loading the Right Pester Version A test file declares the minimum version with `#Requires` (TST-01). That line does not choose @@ -37,8 +40,9 @@ Invoke-Pester -Path ./Tests -FullNameFilter '*Get-ServerHealth*' Pester 6 treats as `Normal`. For coverage, reports, or any other option, pass a configuration object; see -[Pester Configuration Guide](./pester-configuration.md). To run exactly what CI runs, call the gate -script that CI calls; see [CI/CD Integration Guide](./cicd-integration.md#the-gate-script). +[Pester Configuration Guide](./pester-configuration.md). To run exactly what CI runs, call the +script that CI calls: the [runner script](#the-test-runner-script) below, or the gate script in +[CI/CD Integration Guide](./cicd-integration.md#the-gate-script). ### Console Output @@ -56,6 +60,757 @@ A discovery-only run (`Run.SkipRun`) prints `Starting discovery in 4 files.`, Do not read results from this text. Set `Run.PassThru` and use the result object. +## The Test Runner Script + +Keep one standard script, `Tests/Invoke-Tests.ps1`, so that tests run the same way on every machine +and in CI. It takes the type of test and the environment, loads the configuration files of +[Pester Configuration Guide](./pester-configuration.md#settings-for-each-environment), prints a +summary, writes the reports, and applies the gate. + +The script, its helper file, and the settings files are kept in the `Tests` folder, as the layout +in [Pester Test Structure Guide](./test-structure-guide.md#test-directory-layout) shows. A `build` +folder beside `Tests` is the alternative; both are one level below the repository root, so the +script is the same in either. + +```powershell +#requires -Version 7.6 + +<# +.SYNOPSIS +Runs the Pester tests of one type and applies the quality gate. + +.DESCRIPTION +Loads the Pester configuration of the environment, selects the tests of the requested type, runs +them, and prints a summary. Writes a test result report and a metrics file, and on request a +coverage report and an HTML report. Throws when no tests ran, when the Pester result is not +'Passed', or when code coverage is below the configured target. With PassThru the result object is +returned and the last two checks are left to the caller. + +.PARAMETER TestType +Tests to run: Unit, Integration, Performance, Security, or All. + +.PARAMETER Environment +Environment whose configuration file is merged over the standard file. + +.PARAMETER OutputPath +Folder that receives the reports. The default is Tests/Results in the repository. + +.PARAMETER CodeCoverage +Measures code coverage and applies the coverage gate. + +.PARAMETER PassThru +Returns the Pester result object. + +.PARAMETER ShowReport +Writes an HTML report of the run. + +.PARAMETER Tag +Tags to run in place of the tag of the test type. A test that has any one of the tags runs, and +the folder of the test type still limits the run. + +.PARAMETER ExcludeTag +Tags to leave out. + +.PARAMETER Parallel +Experimental. Runs test files at the same time. Ignored for performance and integration tests. + +.PARAMETER ThrottleLimit +Number of files to run at once in a parallel run. 0 uses every processor. + +.PARAMETER NoShuffle +Runs the tests in their declared order. Without it every run is shuffled, except a run of +performance tests: files, blocks, and tests run in a random order. Experimental in Pester. + +.PARAMETER ShuffleSeed +Seed of an earlier shuffled run, to repeat its order. 0 picks a new seed and prints it. + +.EXAMPLE +./Tests/Invoke-Tests.ps1 -TestType Unit + +Runs the unit tests with the development configuration. + +.EXAMPLE +./Tests/Invoke-Tests.ps1 -TestType Unit -Environment CI -CodeCoverage + +Runs the unit tests as CI does, in a shuffled order, with the coverage gate. + +.EXAMPLE +./Tests/Invoke-Tests.ps1 -TestType Unit -ShuffleSeed 852659930 + +Repeats the order of an earlier run, to reproduce a failure that it reported. + +.INPUTS +None. This script does not accept pipeline input. + +.OUTPUTS +None, or the Pester result object when PassThru is given. +#> +[CmdletBinding()] +param( + [ValidateSet('Unit', 'Integration', 'Performance', 'Security', 'All')] + [string]$TestType = 'All', + + [ValidateSet('Development', 'CI', 'Production')] + [string]$Environment = 'Development', + + [string]$OutputPath, + + [switch]$CodeCoverage, + + [switch]$PassThru, + + [switch]$ShowReport, + + [string[]]$Tag = @(), + + [string[]]$ExcludeTag = @(), + + [switch]$Parallel, + + [int]$ThrottleLimit = 0, + + [switch]$NoShuffle, + + [int]$ShuffleSeed = 0 +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$repositoryRoot = Split-Path -Path $PSScriptRoot -Parent +$testRoot = Join-Path -Path $repositoryRoot -ChildPath 'Tests' +if (-not $OutputPath) { + $OutputPath = Join-Path -Path $testRoot -ChildPath 'Results' +} +$null = New-Item -Path $OutputPath -ItemType Directory -Force + +. (Join-Path -Path $PSScriptRoot -ChildPath 'TestRunnerHelpers.ps1') + +Write-Host "Test type: $TestType | Environment: $Environment" -ForegroundColor Green + +$environmentCheck = Test-TestEnvironment -TestDataPath (Join-Path -Path $testRoot -ChildPath 'TestData') +if (-not $environmentCheck.Valid) { + throw "The test environment is not ready. $($environmentCheck.Issues -join ' ')" +} +Import-Module -Name Pester -RequiredVersion 6.2.0 + +$configuration = Get-PesterConfiguration -Environment $Environment -ConfigPath $PSScriptRoot +$configuration.Run.RepoRoot = $repositoryRoot +$configuration.Run.PassThru = $true +$timestamp = Get-Date -Format 'yyyyMMdd-HHmmss' + +# The tests of the requested type +$typeFolders = @( + Get-ChildItem -LiteralPath $testRoot -Directory | + Where-Object { $_.Name -in 'Unit', 'Integration', 'Performance', 'Security' } | + ForEach-Object { $_.FullName } +) +if ($typeFolders.Count -eq 0) { + # A flat Tests folder (MOD-01): every test file is in the run, and the tag selects the type. + $configuration.Run.Path = $testRoot +} elseif ($TestType -eq 'All') { + # The Performance and Security folders exist only where such tests do (TST-43). + $configuration.Run.Path = $typeFolders +} else { + $configuration.Run.Path = Join-Path -Path $testRoot -ChildPath $TestType +} +if ($TestType -ne 'All') { + $configuration.Filter.Tag = $TestType +} +if ($Tag.Count -gt 0) { + # Several tags select a test that has any one of them, so these tags replace the tag of the + # test type. The folder of the type still limits the run. + $configuration.Filter.Tag = $Tag +} +if ($ExcludeTag.Count -gt 0) { + $configuration.Filter.ExcludeTag = $ExcludeTag +} + +# Code coverage +$configuration.CodeCoverage.Enabled = [bool]$CodeCoverage +if ($CodeCoverage) { + $coverageFile = "Coverage-$TestType-$timestamp.xml" + $configuration.CodeCoverage.OutputPath = Join-Path -Path $OutputPath -ChildPath $coverageFile + Write-Host 'Code coverage enabled (profiler-based)' -ForegroundColor Green +} + +# Parallel execution +if ($Parallel) { + if ($TestType -in 'Performance', 'Integration') { + Write-Warning "$TestType tests should not run in parallel; ignoring -Parallel." + } else { + $configuration.Run.Parallel = $true + $configuration.Run.ParallelThrottleLimit = $ThrottleLimit + $limitText = if ($ThrottleLimit -eq 0) { 'all processors' } else { "$ThrottleLimit files" } + Write-Host "Parallel execution enabled (throttle: $limitText)" -ForegroundColor Green + + if ($CodeCoverage) { + # A parallel run collects coverage, but with breakpoints instead of the profiler. + Write-Warning 'Coverage in a parallel run uses breakpoints, not the profiler.' + } + } +} + +# Shuffled order, in every run except a run of performance tests (TST-46). The settings file may +# have set the option either way, so it is set here in both cases. +$isShuffled = -not $NoShuffle -and $TestType -ne 'Performance' +$configuration.Run.Shuffle = $isShuffled +if ($isShuffled) { + if ($ShuffleSeed -ne 0) { + $configuration.Run.ShuffleSeed = $ShuffleSeed + Write-Host "Shuffled execution replaying seed $ShuffleSeed" -ForegroundColor Green + } else { + Write-Host 'Shuffled execution enabled (the seed is printed below)' -ForegroundColor Green + } +} else { + Write-Host 'Shuffled execution is off (declared order)' -ForegroundColor Yellow + if ($ShuffleSeed -ne 0) { + Write-Warning 'This run is not shuffled; ignoring -ShuffleSeed.' + } +} + +# Test result report, with the type and the time in its name +$resultFile = "TestResults-$TestType-$timestamp.xml" +$configuration.TestResult.Enabled = $true +$configuration.TestResult.OutputPath = Join-Path -Path $OutputPath -ChildPath $resultFile + +# Verbosity for the environment +if ($Environment -eq 'CI') { + $configuration.Output.Verbosity = 'Normal' +} else { + $configuration.Output.Verbosity = 'Detailed' +} + +# The relative paths of the configuration files are resolved from the repository root. +Push-Location -LiteralPath $repositoryRoot +try { + $configurationCheck = Test-PesterConfiguration -Configuration $configuration + if (-not $configurationCheck.Valid) { + throw "The Pester configuration is not valid. $($configurationCheck.Issues -join ' ')" + } + + $startTime = Get-Date + Write-Host "Test execution started at $($startTime.ToString('yyyy-MM-dd HH:mm:ss'))" + $result = Invoke-Pester -Configuration $configuration + $duration = (Get-Date) - $startTime +} finally { + Pop-Location +} + +if ($null -eq $result -or $result.TotalCount -eq 0) { + throw 'Pester ran no tests.' +} + +# Summary +$failedColor = if ($result.FailedCount -gt 0) { 'Red' } else { 'Green' } +Write-Host '' +Write-Host 'Test Execution Summary:' -ForegroundColor Green +Write-Host " Environment: $Environment" +Write-Host " Test type: $TestType" +Write-Host " Duration: $($duration.ToString('hh\:mm\:ss'))" +Write-Host " Total tests: $($result.TotalCount)" +Write-Host " Passed: $($result.PassedCount)" -ForegroundColor Green +Write-Host " Failed: $($result.FailedCount)" -ForegroundColor $failedColor +Write-Host " Skipped: $($result.SkippedCount)" -ForegroundColor Yellow +Write-Host " Inconclusive: $($result.InconclusiveCount)" -ForegroundColor Yellow +Write-Host " Not run: $($result.NotRunCount)" + +# Coverage summary. The names are CoveragePercent, CommandsExecutedCount, and +# CommandsAnalyzedCount. The Pester 4 names CoveredPercent and NumberOfCommandsExecuted do not +# exist. +$coveragePercent = $null +if ($CodeCoverage -and $null -ne $result.CodeCoverage) { + $coveragePercent = [math]::Round($result.CodeCoverage.CoveragePercent, 2) + $coverageTarget = $result.CodeCoverage.CoveragePercentTarget + $coverageColor = if ($coveragePercent -ge $coverageTarget) { + 'Green' + } elseif ($coveragePercent -ge ($coverageTarget * 0.75)) { + 'Yellow' + } else { + 'Red' + } + Write-Host " Code Coverage: $coveragePercent% (target $coverageTarget%)" -ForegroundColor $coverageColor + Write-Host " Commands covered: $($result.CodeCoverage.CommandsExecutedCount)" + Write-Host " Commands total: $($result.CodeCoverage.CommandsAnalyzedCount)" + Write-Host " Files analyzed: $($result.CodeCoverage.FilesAnalyzedCount)" +} + +# Performance summary. The Duration of a test is a TimeSpan: use TotalMilliseconds, not the +# object itself. +if ($TestType -in 'Performance', 'All') { + $performanceRoot = Join-Path -Path $testRoot -ChildPath 'Performance' + $performanceTests = @( + $result.Tests | Where-Object { + $_.ScriptBlock.File.StartsWith($performanceRoot, [System.StringComparison]::OrdinalIgnoreCase) + } + ) + if ($performanceTests.Count -gt 0) { + $averageMs = ($performanceTests.Duration.TotalMilliseconds | Measure-Object -Average).Average + Write-Host '' + Write-Host 'Performance Summary:' -ForegroundColor Cyan + Write-Host " Average test duration: $([math]::Round($averageMs, 2))ms" + + $slowTests = $performanceTests | + Where-Object { $_.Duration.TotalSeconds -gt 5 } | + Sort-Object -Property Duration -Descending + foreach ($slowTest in $slowTests) { + $seconds = [math]::Round($slowTest.Duration.TotalSeconds, 2) + Write-Host " Slow test (over 5 s): $($slowTest.ExpandedPath): ${seconds}s" -ForegroundColor Yellow + } + } +} + +# Failed tests, each with its error and a hint +if ($result.FailedCount -gt 0) { + Write-Host '' + Write-Host 'Failed tests:' -ForegroundColor Red + foreach ($test in $result.Failed) { + $testError = $test.ErrorRecord[0] + Write-Host " - $($test.ExpandedPath)" -ForegroundColor Red + Write-Host " Error: $($testError.Exception.Message)" -ForegroundColor Red + + $hint = Get-TroubleshootingHint -TestName $test.Name -ErrorRecord $testError + if ($hint) { + Write-Host " Hint: $hint" -ForegroundColor Cyan + } + } +} + +# Files that failed discovery or setup. Such a file adds nothing to FailedCount, so a suite can +# report 0 failed tests while whole files never ran. +if ($result.FailedContainersCount -gt 0) { + Write-Host '' + Write-Host 'Failed files (discovery or setup errors):' -ForegroundColor Red + foreach ($container in $result.FailedContainers) { + Write-Host " - $($container.Item)" -ForegroundColor Red + foreach ($containerError in $container.ErrorRecord) { + Write-Host " $($containerError.Exception.Message)" -ForegroundColor Red + } + } +} + +# HTML report +if ($ShowReport) { + $reportPath = Join-Path -Path $OutputPath -ChildPath "TestReport-$TestType-$timestamp.html" + New-TestReport -TestResult $result -OutputPath $reportPath -CodeCoverage:$CodeCoverage + Write-Host "Test report written: $reportPath" -ForegroundColor Green +} + +# Metrics for trending. The seed is recorded because a shuffled failure can be repeated only +# while the seed outlives the console log. +$metrics = [ordered]@{ + TestType = $TestType + Environment = $Environment + Timestamp = Get-Date -Format 'o' + Duration = $duration.TotalSeconds + TotalTests = $result.TotalCount + PassedTests = $result.PassedCount + FailedTests = $result.FailedCount + SkippedTests = $result.SkippedCount + CodeCoverage = $coveragePercent + Parallel = [bool]$configuration.Run.Parallel.Value + Shuffle = [bool]$configuration.Run.Shuffle.Value + ShuffleSeed = $result.Configuration.Run.ShuffleSeed.Value + PesterVersion = (Get-Module -Name Pester).Version.ToString() + PowerShellVersion = $PSVersionTable.PSVersion.ToString() + Platform = [System.Environment]::OSVersion.Platform.ToString() +} +$metricsPath = Join-Path -Path $OutputPath -ChildPath "TestMetrics-$TestType-$timestamp.json" +$metrics | ConvertTo-Json | Set-Content -LiteralPath $metricsPath -Encoding utf8 + +if ($Environment -eq 'CI') { + # GitHub Actions step outputs. The ::set-output command is deprecated and the runner warns: write + # to the file that GITHUB_OUTPUT names. + if ($env:GITHUB_OUTPUT) { + $outputLines = @( + "total-tests=$($result.TotalCount)" + "passed-tests=$($result.PassedCount)" + "failed-tests=$($result.FailedCount)" + if ($null -ne $coveragePercent) { + "code-coverage=$coveragePercent" + } + ) + Add-Content -LiteralPath $env:GITHUB_OUTPUT -Value $outputLines -Encoding utf8 + } + + # GitHub Actions job summary + if ($env:GITHUB_STEP_SUMMARY) { + $summaryLines = @( + "## Test Results - $TestType" + '' + '| Metric | Value |' + '| --- | --- |' + "| Total | $($result.TotalCount) |" + "| Passed | $($result.PassedCount) |" + "| Failed | $($result.FailedCount) |" + "| Skipped | $($result.SkippedCount) |" + if ($null -ne $coveragePercent) { + "| Code coverage | $coveragePercent% (target $coverageTarget%) |" + } + '' + ) + Add-Content -LiteralPath $env:GITHUB_STEP_SUMMARY -Value $summaryLines -Encoding utf8 + } + + # Azure Pipelines variables + if ($env:BUILD_BUILDID) { + Write-Host "##vso[task.setvariable variable=TotalTests]$($result.TotalCount)" + Write-Host "##vso[task.setvariable variable=PassedTests]$($result.PassedCount)" + Write-Host "##vso[task.setvariable variable=FailedTests]$($result.FailedCount)" + if ($null -ne $coveragePercent) { + Write-Host "##vso[task.setvariable variable=CodeCoverage]$coveragePercent" + } + } +} + +if ($PassThru) { + $result + return +} + +# The gate. Result also covers a failed BeforeAll or AfterAll and a file that failed discovery. +if ($result.Result -ne 'Passed') { + $counts = $result.FailedCount, $result.FailedBlocksCount, $result.FailedContainersCount + $summary = 'Failed tests: {0}. Failed blocks: {1}. Failed files: {2}.' -f $counts + if ($configuration.Run.Shuffle.Value) { + $seed = $result.Configuration.Run.ShuffleSeed.Value + $summary = "$summary Replay with Run.ShuffleSeed = $seed." + } + throw "Pester result is '$($result.Result)'. $summary" +} + +# CoveragePercentTarget sets the reported target and does not fail the run, so the script +# compares the figure and lists the uncovered commands for each file. +if ($null -ne $coveragePercent -and $coveragePercent -lt $coverageTarget) { + Write-Host 'Uncovered commands:' + foreach ($file in $result.CodeCoverage.CommandsMissed | Group-Object -Property File) { + Write-Host " $($file.Name): $($file.Count) command(s)" + if ($env:GITHUB_ACTIONS) { + Write-Host "::warning file=$($file.Name)::$($file.Count) uncovered command(s)" + } + } + throw "Code coverage is $coveragePercent percent. The target is $coverageTarget." +} + +Write-Host 'All tests passed.' -ForegroundColor Green +``` + +Notes on the script: + +- It fails by throwing (CIC-05), and the gate tests `Result`, not `FailedCount` alone (TST-42). +- `-Tag` replaces the tag of the test type, because several tags select a test that has any one of + them and adding a tag would widen the run. The folder of the type still limits it, so + `-TestType Unit -Tag CrossPlatform` runs the tests under `Tests/Unit` that are tagged + `CrossPlatform`. Use `-ExcludeTag` to narrow a selection further. +- A flat `Tests` folder (MOD-01) has no folder for a test type. The script then runs every test + file in `Tests`, and the tag of the test type alone selects the tests, so `-Tag` decides the + selection by itself there. +- Every report has the test type and the time in its name, in `Tests/Results`, so that runs do not + overwrite each other and one artifact pattern collects them: `Tests/Results/TestResults*.xml`, + `Tests/Results/Coverage*.xml`, and `Tests/Results/TestMetrics*.json`. Do not commit the folder. +- `Run.Parallel` and `Run.Shuffle` are experimental. The script turns `Run.Parallel` on only when + the caller passes `-Parallel`. It shuffles every run except a run of performance tests (TST-46); + `-NoShuffle` runs the tests in their declared order, for example to compare with a shuffled + failure. +- The script targets PowerShell 7.6. For a project that declares Windows PowerShell 5.1 support, + change the first line to `#requires -Version 5.1`. The script and its helper functions were run + under Windows PowerShell 5.1 with that line, with and without `-Parallel`. + +### Quick Commands + +```powershell +# Unit tests during development +./Tests/Invoke-Tests.ps1 -TestType Unit -Environment Development + +# Unit tests with coverage and an HTML report +./Tests/Invoke-Tests.ps1 -TestType Unit -CodeCoverage -ShowReport + +# Tests with a tag, leaving out the slow ones +./Tests/Invoke-Tests.ps1 -Tag 'Critical' -ExcludeTag 'Slow' + +# Fast feedback: unit tests in parallel, without coverage +./Tests/Invoke-Tests.ps1 -TestType Unit -Parallel + +# Repeat the order of a run that failed, then compare with the declared order +./Tests/Invoke-Tests.ps1 -TestType Unit -ShuffleSeed 852659930 +./Tests/Invoke-Tests.ps1 -TestType Unit -NoShuffle + +# What CI runs +./Tests/Invoke-Tests.ps1 -TestType All -Environment CI -CodeCoverage +./Tests/Invoke-Tests.ps1 -TestType Performance -Environment CI +./Tests/Invoke-Tests.ps1 -TestType Security -Environment CI +``` + +### Checking the Environment First + +The runner validates the environment before it loads Pester: the PowerShell version, the pinned +Pester version, and the test data folder. The configuration is validated by +`Test-PesterConfiguration`; see +[Pester Configuration Guide](./pester-configuration.md#validating-a-configuration-before-the-run). + +```powershell +# Tests/TestRunnerHelpers.ps1 +function Test-TestEnvironment { + [CmdletBinding()] + [OutputType('TestRunner.Check')] + param( + [string]$TestDataPath + ) + + $psVersion = $PSVersionTable.PSVersion + $pesterVersions = @((Get-Module -Name Pester -ListAvailable).Version) + + $issues = @( + # Pester 6 runs on Windows PowerShell 5.1 and on PowerShell 7.4 or later. + if ($psVersion.Major -eq 5 -and $psVersion.Minor -lt 1) { + "Windows PowerShell 5.1 is required (found $psVersion)." + } elseif ($psVersion.Major -eq 6 -or ($psVersion.Major -eq 7 -and $psVersion.Minor -lt 4)) { + "PowerShell 7.4 or later is required (found $psVersion)." + } + + if ($pesterVersions.Count -eq 0) { + 'Required module missing: Pester.' + } elseif ($pesterVersions -notcontains [version]'6.2.0') { + "Pester 6.2.0 is required (found $($pesterVersions -join ', '))." + } + + if ($TestDataPath -and -not (Test-Path -LiteralPath $TestDataPath)) { + "Test data folder missing: $TestDataPath" + } + ) + + [PSCustomObject]@{ + PSTypeName = 'TestRunner.Check' + Valid = $issues.Count -eq 0 + Issues = $issues + } +} +``` + +### Failure Hints + +For each failed test the runner prints the error and, where the message is a known one, a hint. Add +the messages that a project meets often. + +```powershell +# Tests/TestRunnerHelpers.ps1 +function Get-TroubleshootingHint { + [CmdletBinding()] + [OutputType([string])] + param( + [Parameter(Mandatory)] + [string]$TestName, + + # Not named $Error: that name belongs to an automatic variable (STY-13). + [Parameter(Mandatory)] + [System.Management.Automation.ErrorRecord]$ErrorRecord + ) + + $hints = [ordered]@{ + 'Connection' = 'Check network connectivity and firewall settings.' + 'Authentication' = 'Verify credentials and permissions.' + 'Permission' = 'Run as administrator or check file permissions.' + 'Memory' = 'Increase available memory or optimize the code.' + 'Timeout' = 'Increase timeout values or optimize performance.' + 'not recognized' = 'Command not found. In Pester 6 each test file imports its own modules.' + 'ForEach' = 'An empty -ForEach fails discovery in Pester 6. Build test data in BeforeDiscovery.' + 'Assert-Mock' = 'Assert-MockCalled was removed in Pester 6. Use Should -Invoke or Should-Invoke.' + 'to measure or a' = 'Pipe the script block, not its result, to Should-BeFasterThan.' + } + + $message = $ErrorRecord.Exception.Message + foreach ($keyword in $hints.Keys) { + $pattern = [regex]::Escape($keyword) + if ($message -match $pattern -or $TestName -match $pattern) { + return $hints[$keyword] + } + } +} +``` + +The Pester messages behind the last four hints are explained under +[Messages and Their Causes](#messages-and-their-causes). + +### HTML Report + +`-ShowReport` writes an HTML report of the run. Test names and error messages often contain `<`, +`>`, and `&`, which corrupt the report unless they are encoded. + +```powershell +# Tests/TestRunnerHelpers.ps1 +function New-TestReport { + [CmdletBinding(SupportsShouldProcess)] + param( + [Parameter(Mandatory)] + [object]$TestResult, + + [Parameter(Mandatory)] + [string]$OutputPath, + + [switch]$CodeCoverage + ) + + $rows = foreach ($test in $TestResult.Tests) { + $rowClass = switch ($test.Result) { + 'Passed' { 'test-passed' } + 'Failed' { 'test-failed' } + default { '' } + } + $name = [System.Net.WebUtility]::HtmlEncode($test.ExpandedPath) + $message = '' + if ($test.ErrorRecord.Count -gt 0) { + $message = [System.Net.WebUtility]::HtmlEncode($test.ErrorRecord[0].Exception.Message) + } + $milliseconds = [math]::Round($test.Duration.TotalMilliseconds, 2) + + $rowFormat = ' {1}{2}{3}ms{4}' + $rowFormat -f $rowClass, $name, $test.Result, $milliseconds, $message + } + + $coverageRow = '' + if ($CodeCoverage -and $null -ne $TestResult.CodeCoverage) { + $percent = [math]::Round($TestResult.CodeCoverage.CoveragePercent, 2) + $coverageRow = "

Code Coverage: $percent%

" + } + + $html = @" + + + + + PowerShell Test Report + + + +

PowerShell Test Report

+
+

Summary

+

Total Tests: $($TestResult.TotalCount)

+

Passed: $($TestResult.PassedCount)

+

Failed: $($TestResult.FailedCount)

+

Skipped: $($TestResult.SkippedCount)

+

Duration: $($TestResult.Duration)

+ $coverageRow +
+

Test Results

+ + +$($rows -join [Environment]::NewLine) +
Test NameResultDurationError Message
+ + +"@ + + if ($PSCmdlet.ShouldProcess($OutputPath, 'Write the HTML test report')) { + try { + Set-Content -LiteralPath $OutputPath -Value $html -Encoding utf8 -ErrorAction Stop + } catch { + $PSCmdlet.ThrowTerminatingError($_) + } + } +} +``` + +The function encodes with `[System.Net.WebUtility]::HtmlEncode`, which needs no assembly to be +loaded. `[System.Web.HttpUtility]::HtmlEncode` gives the same text but, measured on Windows +PowerShell 5.1, fails with `Unable to find type [System.Web.HttpUtility]` until +`Add-Type -AssemblyName System.Web` has run. On PowerShell 7.6 it worked without that line. + +### Metrics for Trending + +Each run writes its counts, the seed, and the versions to `TestMetrics--