diff --git a/DECISIONS.md b/DECISIONS.md index 486dc5f..63a75c1 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -706,9 +706,16 @@ image or a transcript is not something to read in a source file. - **Its own exit code, `Precondition` (4).** It is neither a failed journey (1) nor a usage error (2), so a workflow can tell "the deploy is not live" from "the site is wrong". Nothing runs, `--retries` does not apply, and the run still reports: the reason on stderr, a failed `preconditions/wait` case in JUnit and the step summary, and an error annotation on GitHub Actions. -- **Not applied when every journey is skipped**, like Chrome itself: a run that checks nothing needs no site. +- **They come before the journeys are planned.** A deploy that has just landed can change the flags and the build type that `@requires` + and `feature` blocks are judged against, so `JourneyRunOptions.RefreshAfterPreconditions` lets the caller read them again once the + wait is met, and the plan, the skips and the run use the refreshed options. The wait therefore also happens when every journey + would be skipped; Chrome starts for it. +- **The flags are parsed here, once.** `JourneyWait.Parse` (`JourneyWaitFlags` in, `JourneyWaitParse` out) is what `snap journey run` + and any tool on `JourneyConsoleRunner` call, so the syntax cannot drift between them. `==`, `=~` and `=*` are found outside quotes + and brackets, and the last `@` outside them starts the attribute: `a[href='mailto:docs@elastic.co']==Contact` is a selector + with no attribute. A problem is plain text naming the flag; the caller adds its own prefix. - **`--wait-for` is `target==value`, not `target=value`.** A CSS selector may hold `=`, `~=` and `*=` (`meta[name='docs:build']`, - `a[href*='x']`), but never `==`, `=~` or `=*`, so those three are the operators and nothing needs escaping. `header:` marks a + `a[href*='x']`), and quoted text may hold anything, so `==`, `=~` and `=*` outside quotes and brackets are the operators. `header:` marks a response header, `@attribute` an attribute, and no `@` means the element's text. - **`compare` waits for the target only.** The baseline is another release, so the value that proves the target deployed is not the baseline's. diff --git a/docs/formats/browser.md b/docs/formats/browser.md index 58f569b..390eb35 100644 --- a/docs/formats/browser.md +++ b/docs/formats/browser.md @@ -257,7 +257,7 @@ are files, not part of the transcript: pixels differ between machines. | `ReadOnly`, `WritableUrls` | `--read-only`, `--writable` | refuse writes, except to these URL globs | | `Viewport` | `--viewport` | the size every journey starts with; Chrome's default is a phone layout on a responsive site | | `UserAgent`, `Headers` | `--user-agent`, `--header` | what every page of the run sends, such as the identity a site uses to tell a monitor from a visitor | -| `Preconditions` | `--wait-for`, `--wait-path`, `--wait-timeout`, `--wait-interval`, `--wait-retries` | wait for a response header or an element value before any journey runs; unmet ends the run with exit code 4 (`JourneyExitCode.Precondition`) | +| `Preconditions` | `--wait-for`, `--wait-path`, `--wait-timeout`, `--wait-interval`, `--wait-retries` | wait for a response header or an element value before any journey runs; unmet ends the run with exit code 4 (`JourneyExitCode.Precondition`). A tool parses the flags with `JourneyWait.Parse` and can re-read the site's flags once the wait is met with `RefreshAfterPreconditions` | | `Retries` | `--retries` | run a journey that fails again this many times; one that passes on a retry passes the run and is reported as flaky (`JourneyOutcome.Flaky`, `EarlierAttempts`) | | `ScreenshotOnFailure` | `--screenshot-on-failure` | take a screenshot of the viewport when a step fails (`JourneyFailure.Screenshot`), written to `failure.png` in the artifacts folder | | `OnStep` | | called before each step, for a progress display | diff --git a/docs/workflow/snap.md b/docs/workflow/snap.md index 591c577..325fef4 100644 --- a/docs/workflow/snap.md +++ b/docs/workflow/snap.md @@ -104,7 +104,7 @@ with exit code 4 and the reason, instead of failing every journey. A check is on --wait-for "header:Server=~^Amazon" # =~ matches a regular expression, =* contains ``` -The operators `==`, `=~` and `=*` never occur in a CSS selector, so a selector needs no escaping. `--wait-path` is the page +The operators `==`, `=~` and `=*` are read outside quotes and brackets, and the last `@` outside them starts the attribute, so a selector such as `a[href='mailto:docs@elastic.co']` needs no escaping. `--wait-path` is the page the checks look at, resolved against `--base-url` (default `./`). `--wait-timeout` (default `10s`; also `500ms`, `2m`) bounds the wait in all and `--wait-interval` (`500ms`) spaces the looks. `--wait-retries` (default 2) is how many looks that fail, because the page did not load or the connection dropped, are tried again before the wait gives up; a look that works and shows another diff --git a/src/Nullean.Snapshots.Browser.Console/JourneyConsoleRunner.cs b/src/Nullean.Snapshots.Browser.Console/JourneyConsoleRunner.cs index 2c0fcde..43549a2 100644 --- a/src/Nullean.Snapshots.Browser.Console/JourneyConsoleRunner.cs +++ b/src/Nullean.Snapshots.Browser.Console/JourneyConsoleRunner.cs @@ -54,29 +54,34 @@ CancellationToken cancellationToken ) { var clock = Stopwatch.StartNew(); - var plan = options.Files.Select(f => Plan(f, options)).ToList(); - reporter.Planned([.. plan.Where(p => p.Skip is not null).Select(p => (p.File, p.Skip!))], plan.Count(p => p.Journey is not null && p.Skip is null)); - - if (plan.Any(p => p.Journey is not null && p.Skip is null)) + var browserStarted = false; + // What the site must show comes first, and before the journeys are planned: a deploy that has just landed can change the + // flags and build type they are judged against, so the caller gets to read them again once the site is as expected. + if (options.Preconditions.Count > 0) { - try - { - await executor.EnsureReadyAsync(cancellationToken).ConfigureAwait(false); - } - catch (BrowserLaunchException e) - { - await error.WriteLineAsync(("Error: " + e.Message).AsMemory(), CancellationToken.None).ConfigureAwait(false); + if (!await StartBrowserAsync(executor, error, cancellationToken).ConfigureAwait(false)) return JourneyExitCode.Browser; - } - + browserStarted = true; if (await UnmetPreconditionAsync(options, executor, reporter, cancellationToken).ConfigureAwait(false) is { } unmet) { await error.WriteLineAsync(("Error: " + unmet).AsMemory(), CancellationToken.None).ConfigureAwait(false); WritePreconditionReports(options, output, unmet, clock.Elapsed); return JourneyExitCode.Precondition; } + + if (options.RefreshAfterPreconditions is { } refresh) + options = await refresh(options, cancellationToken).ConfigureAwait(false); } + var plan = options.Files.Select(f => Plan(f, options)).ToList(); + reporter.Planned([.. plan.Where(p => p.Skip is not null).Select(p => (p.File, p.Skip!))], plan.Count(p => p.Journey is not null && p.Skip is null)); + + if ( + !browserStarted && plan.Any(p => p.Journey is not null && p.Skip is null) && + !await StartBrowserAsync(executor, error, cancellationToken).ConfigureAwait(false) + ) + return JourneyExitCode.Browser; + using var gate = new SemaphoreSlim(Math.Max(1, options.Parallel)); var launched = new LaunchFailure(); var tasks = plan.Select(item => RunOneAsync(item, options, executor, reporter, gate, launched, cancellationToken)); @@ -104,6 +109,21 @@ CancellationToken cancellationToken return ExitCodeOf(outcomes, options); } + // Starts Chrome; false after writing why when it cannot start. + private static async Task StartBrowserAsync(IJourneyExecutor executor, TextWriter error, CancellationToken cancellationToken) + { + try + { + await executor.EnsureReadyAsync(cancellationToken).ConfigureAwait(false); + return true; + } + catch (BrowserLaunchException e) + { + await error.WriteLineAsync(("Error: " + e.Message).AsMemory(), CancellationToken.None).ConfigureAwait(false); + return false; + } + } + // The message of the first precondition that was not met; null when all were. private static async Task UnmetPreconditionAsync( JourneyRunOptions options, diff --git a/src/Nullean.Snapshots.Browser.Console/JourneyRun.cs b/src/Nullean.Snapshots.Browser.Console/JourneyRun.cs index 4caf50a..917e459 100644 --- a/src/Nullean.Snapshots.Browser.Console/JourneyRun.cs +++ b/src/Nullean.Snapshots.Browser.Console/JourneyRun.cs @@ -75,12 +75,19 @@ public sealed class JourneyRunOptions /// /// What the site must show before any journey runs, such as the build a deploy just shipped: waited for one after the other, in - /// the shared Chrome, each within its own timeout and retries. When one is not met nothing runs: the run writes the reason and - /// ends with , and gets a failed - /// preconditions/wait case. Not applied when every journey is skipped, and never retried by . + /// the shared Chrome, each within its own timeout and retries. They come first, before the journeys are planned. When one is not + /// met nothing runs: the run writes the reason and ends with , and + /// gets a failed preconditions/wait case. Never retried by . /// public IReadOnlyList Preconditions { get; init; } = []; + /// + /// Called once every precondition is met and before the journeys are planned, with these options, to return the options the run + /// uses: a deploy that has just landed can change the flags and build type that @requires and feature blocks are + /// judged against, so a caller that read them before the wait reads them again here. Not called without preconditions. + /// + public Func>? RefreshAfterPreconditions { get; init; } + /// The enabled feature flags of the site under test. public IReadOnlySet Features { get; init; } = new HashSet(StringComparer.Ordinal); diff --git a/src/Nullean.Snapshots.Browser.Console/JourneyWait.cs b/src/Nullean.Snapshots.Browser.Console/JourneyWait.cs new file mode 100644 index 0000000..3507fab --- /dev/null +++ b/src/Nullean.Snapshots.Browser.Console/JourneyWait.cs @@ -0,0 +1,203 @@ +using System.Globalization; +using Nullean.Cdp; +using Nullean.Cdp.Journeys; + +namespace Nullean.Snapshots.Browser.Console; + +/// +/// The command-line form of , as the flags of snap journey run and any tool built on +/// take them. Raw text, as typed: checks it. +/// +public sealed record JourneyWaitFlags +{ + /// + /// The checks, each header:Name==value (a response header), selector==value (an element's text) or + /// selector@attribute==value (an attribute). =~ in place of == matches a regular expression and =* contains. + /// + public IReadOnlyList Checks { get; init; } = []; + + /// The page the checks look at, resolved against the base URL. Default: ./. + public string? Path { get; init; } + + /// How long the wait lasts in all: 10s, 500ms, 2m, or a number of seconds. Default: 10 seconds. + public string? Timeout { get; init; } + + /// How long to wait between looks, in the same form. Default: 500 milliseconds. + public string? Interval { get; init; } + + /// How many looks that fail are tried again. Default: . + public int? Retries { get; init; } +} + +/// What made of the flags: the preconditions, or what is wrong with them. +/// One per check; empty when there is a or when no check was given. +/// Plain text naming the flag at fault, without a prefix; null when the flags are valid. +public sealed record JourneyWaitParse(IReadOnlyList Preconditions, string? Problem = null) +{ + /// The flags are valid. + public bool Valid => Problem is null; +} + +/// Reads the wait flags of a command line into s. +public static class JourneyWait +{ + private static readonly (string Token, TextMatch Match)[] Operators = [("==", TextMatch.Exact), ("=~", TextMatch.Regex), ("=*", TextMatch.Contains)]; + + /// + /// Reads . The operators ==, =~ and =* are found outside quotes and brackets, and the + /// last @ outside them starts the attribute, so a selector such as meta[name='docs:build'] or + /// a[href='mailto:docs@elastic.co'] needs no escaping. Timeout, interval and retries apply to every check. + /// + public static JourneyWaitParse Parse(JourneyWaitFlags flags) + { + ArgumentNullException.ThrowIfNull(flags); + if (flags.Checks.Count == 0) + return flags.Path is not null || flags.Timeout is not null || flags.Interval is not null || flags.Retries is not null + ? new JourneyWaitParse([], "--wait-path, --wait-timeout, --wait-interval and --wait-retries describe a --wait-for; give one.") + : new JourneyWaitParse([]); + + if ( + !TryDuration("--wait-timeout", flags.Timeout, TimeSpan.FromSeconds(10), out var timeout, out var problem) || + !TryDuration("--wait-interval", flags.Interval, TimeSpan.FromMilliseconds(500), out var interval, out problem) + ) + return new JourneyWaitParse([], problem); + if (flags.Retries is < 0) + return new JourneyWaitParse([], $"--wait-retries {flags.Retries} is not zero or more."); + + var preconditions = new List(); + foreach (var check in flags.Checks) + { + if (!TryCheck(check, out var parsed, out problem)) + return new JourneyWaitParse([], problem); + preconditions.Add(parsed with { Path = flags.Path ?? "./", Timeout = timeout, Interval = interval, Retries = flags.Retries ?? parsed.Retries }); + } + + return new JourneyWaitParse(preconditions); + } + + private static bool TryCheck(string value, out JourneyPrecondition precondition, out string problem) + { + problem = string.Empty; + precondition = null!; + var (index, length, match) = FindOperator(value); + var target = index < 0 ? string.Empty : value[..index]; + var expected = index < 0 ? string.Empty : value[(index + length)..]; + if (index < 0 || target.Length == 0 || expected.Length == 0) + { + problem = + $"--wait-for '{value}' is not a check: use header:Name==value, selector==value or selector@attribute==value ('=~' matches a regular expression, '=*' contains)."; + return false; + } + + if (target.StartsWith("header:", StringComparison.OrdinalIgnoreCase)) + { + var name = target["header:".Length..]; + if (name.Length == 0) + { + problem = $"--wait-for '{value}' names no header."; + return false; + } + + precondition = new JourneyPrecondition { Path = "./", Header = name, Expected = expected, Match = match }; + return true; + } + + string? attribute = null; + if (LastTopLevel(target, '@') is var at and >= 0) + { + attribute = target[(at + 1)..]; + target = target[..at]; + } + + if (target.Length == 0 || attribute is { Length: 0 }) + { + problem = $"--wait-for '{value}' is not a check: the selector or the attribute is empty."; + return false; + } + + precondition = new JourneyPrecondition { Path = "./", Selector = target, Attribute = attribute, Expected = expected, Match = match }; + return true; + } + + // The first operator outside quotes and brackets, which is where the target ends and the expected value starts. + private static (int Index, int Length, TextMatch Match) FindOperator(string value) + { + var scan = new Scan(); + for (var i = 0; i < value.Length; i++) + { + if (scan.Outside(value[i])) + { + foreach (var (token, match) in Operators) + if (string.CompareOrdinal(value, i, token, 0, token.Length) == 0) + return (i, token.Length, match); + } + } + + return (-1, 0, TextMatch.Exact); + } + + private static int LastTopLevel(string text, char wanted) + { + var scan = new Scan(); + var last = -1; + for (var i = 0; i < text.Length; i++) + { + if (scan.Outside(text[i]) && text[i] == wanted) + last = i; + } + + return last; + } + + // Tracks whether the scan is inside a quoted string or a [bracket], as in a CSS attribute selector. + private struct Scan + { + private char _quote; + private int _depth; + + // Whether this character sits outside quotes and brackets; updates the state for the next one. + public bool Outside(char c) + { + if (_quote != 0) + { + if (c == _quote) + _quote = '\0'; + return false; + } + + switch (c) + { + case '\'' or '"': + _quote = c; + return false; + case '[': + _depth++; + return false; + case ']' when _depth > 0: + _depth--; + return false; + default: + return _depth == 0; + } + } + } + + private static bool TryDuration(string flag, string? value, TimeSpan fallback, out TimeSpan duration, out string problem) + { + problem = string.Empty; + duration = fallback; + if (value is null) + return true; + var (digits, unit) = value.EndsWith("ms", StringComparison.Ordinal) + ? (value[..^2], 0.001) + : value.EndsWith('s') ? (value[..^1], 1.0) : value.EndsWith('m') ? (value[..^1], 60.0) : (value, 1.0); + if (double.TryParse(digits, NumberStyles.Float, CultureInfo.InvariantCulture, out var number) && number > 0) + { + duration = TimeSpan.FromSeconds(number * unit); + return true; + } + + problem = $"{flag} '{value}' is not a duration: use 10s, 500ms or 2m."; + return false; + } +} diff --git a/src/Nullean.Snapshots.Cli/JourneyCli.cs b/src/Nullean.Snapshots.Cli/JourneyCli.cs index c184987..6ebc3c3 100644 --- a/src/Nullean.Snapshots.Cli/JourneyCli.cs +++ b/src/Nullean.Snapshots.Cli/JourneyCli.cs @@ -1,7 +1,6 @@ using System.IO.Abstractions; using System.Text.RegularExpressions; using Nullean.Argh; -using Nullean.Cdp; using Nullean.Cdp.Journeys; using Nullean.Snapshots.Browser; using Nullean.Snapshots.Browser.Console; @@ -530,129 +529,20 @@ out var width return false; } - /// - /// Reads each --wait-for: header:Name==value for a response header, selector==value for an element's text and - /// selector@attribute==value for an attribute. =~ in place of == matches a regular expression and =* contains. - /// The operators never occur in a CSS selector, so a selector such as meta[name='docs:build'] needs no escaping. - /// + /// Reads --wait-for and its options with ; a problem is a usage error. internal static bool TryParseWaits(JourneyRunArgs args, out List preconditions, out string problem) { - problem = string.Empty; - preconditions = []; - if (args.WaitFor.Count == 0) - { - if (args.WaitPath is not null || args.WaitTimeout is not null || args.WaitInterval is not null || args.WaitRetries is not null) - { - problem = "Error: --wait-path, --wait-timeout, --wait-interval and --wait-retries describe a --wait-for; give one."; - return false; - } - - return true; - } - - if ( - !TryParseDuration("--wait-timeout", args.WaitTimeout, TimeSpan.FromSeconds(10), out var timeout, out problem) || - !TryParseDuration("--wait-interval", args.WaitInterval, TimeSpan.FromMilliseconds(500), out var interval, out problem) - ) - return false; - if (args.WaitRetries is < 0) - { - problem = $"Error: --wait-retries {args.WaitRetries} is not zero or more."; - return false; - } - - foreach (var value in args.WaitFor) - { - if (!TryParseWait(value, out var parsed, out problem)) - return false; - preconditions.Add(parsed with - { - Path = args.WaitPath ?? "./", - Timeout = timeout, - Interval = interval, - Retries = args.WaitRetries ?? parsed.Retries, - }); - } - - return true; - } - - private static bool TryParseWait(string value, out JourneyPrecondition precondition, out string problem) - { - problem = string.Empty; - precondition = null!; - var (index, length, match) = FindOperator(value); - var target = index < 0 ? string.Empty : value[..index]; - var expected = index < 0 ? string.Empty : value[(index + length)..]; - if (index < 0 || target.Length == 0 || expected.Length == 0) - { - problem = - $"Error: --wait-for '{value}' is not a wait: use header:Name==value, selector==value or selector@attribute==value ('=~' matches a regular expression, '=*' contains)."; - return false; - } - - if (target.StartsWith("header:", StringComparison.OrdinalIgnoreCase)) - { - var name = target["header:".Length..]; - if (name.Length == 0) - { - problem = $"Error: --wait-for '{value}' names no header."; - return false; - } - - precondition = new JourneyPrecondition { Path = "./", Header = name, Expected = expected, Match = match }; - return true; - } - - var at = target.LastIndexOf('@'); - string? attribute = null; - if (at >= 0) - { - attribute = target[(at + 1)..]; - target = target[..at]; - } - - if (target.Length == 0 || attribute is { Length: 0 }) - { - problem = $"Error: --wait-for '{value}' is not a wait: the selector or the attribute is empty."; - return false; - } - - precondition = new JourneyPrecondition { Path = "./", Selector = target, Attribute = attribute, Expected = expected, Match = match }; - return true; - } - - private static (int Index, int Length, TextMatch Match) FindOperator(string value) - { - var best = (Index: -1, Length: 0, Match: TextMatch.Exact); - foreach (var (token, match) in new[] { ("==", TextMatch.Exact), ("=~", TextMatch.Regex), ("=*", TextMatch.Contains) }) - { - var at = value.IndexOf(token, StringComparison.Ordinal); - if (at >= 0 && (best.Index < 0 || at < best.Index)) - best = (at, token.Length, match); - } - - return best; - } - - /// Reads a duration: 10s, 500ms, 2m, or a bare number of seconds. - internal static bool TryParseDuration(string flag, string? value, TimeSpan fallback, out TimeSpan duration, out string problem) - { - problem = string.Empty; - duration = fallback; - if (value is null) - return true; - var (digits, unit) = value.EndsWith("ms", StringComparison.Ordinal) - ? (value[..^2], 0.001) - : value.EndsWith('s') ? (value[..^1], 1.0) : value.EndsWith('m') ? (value[..^1], 60.0) : (value, 1.0); - if (double.TryParse(digits, System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out var number) && number > 0) - { - duration = TimeSpan.FromSeconds(number * unit); - return true; - } - - problem = $"Error: {flag} '{value}' is not a duration: use 10s, 500ms or 2m."; - return false; + var parsed = JourneyWait.Parse(new JourneyWaitFlags + { + Checks = args.WaitFor, + Path = args.WaitPath, + Timeout = args.WaitTimeout, + Interval = args.WaitInterval, + Retries = args.WaitRetries, + }); + preconditions = [.. parsed.Preconditions]; + problem = parsed.Problem is null ? string.Empty : "Error: " + parsed.Problem; + return parsed.Valid; } /// Reads each --header: "Name: value". diff --git a/tests/Nullean.Snapshots.Browser.Tests/JourneyConsoleRunnerTests.cs b/tests/Nullean.Snapshots.Browser.Tests/JourneyConsoleRunnerTests.cs index dbd866e..5ba7d9e 100644 --- a/tests/Nullean.Snapshots.Browser.Tests/JourneyConsoleRunnerTests.cs +++ b/tests/Nullean.Snapshots.Browser.Tests/JourneyConsoleRunnerTests.cs @@ -330,27 +330,83 @@ public async Task PreconditionsAreWaitedForInOrderAndTheFirstUnmetOneStopsTheRun } [Test] - public async Task NoPreconditionIsWaitedForWhenEveryJourneyIsSkipped() + public async Task OptionsAreRefreshedOnceThePreconditionsAreMetSoTheJourneysAreJudgedAgainstTheNewBuild() { using var folder = new Folder(); - var options = new JourneyRunOptions + // The deploy that the wait is for turns a flag on: before it the journey is skipped, after it the journey runs. + var stale = new JourneyRunOptions { - Files = [folder.Add("a/needs", "# @requires build-type:isolated\nopen /")], - BuildType = "assembler", + Files = [folder.Add("a/needs", "# @requires navigation-preview\nopen /")], Preconditions = [Build], - AllowAllSkipped = true, GitHubActions = false, + RefreshAfterPreconditions = + (current, _) => Task.FromResult(new JourneyRunOptions + { + Files = current.Files, + Features = new HashSet { "navigation-preview" }, + GitHubActions = false + }), }; var waited = false; - var (exit, _, _) = await RunAsync(options, wait: _ => + var (exit, output, _) = await RunAsync(stale, wait: _ => { waited = true; - return new PreconditionResult(false, "never", 1, TimeSpan.Zero); + return new PreconditionResult(true, "ok", 1, TimeSpan.Zero); }); exit.Should().Be(JourneyExitCode.Ok); - waited.Should().BeFalse(); + waited.Should().BeTrue(); + output.Should().Contain("ok a/needs").And.NotContain("skipped a/needs"); + } + + [Test] + public async Task OptionsAreNotRefreshedWhenAPreconditionIsNotMetOrThereIsNone() + { + using var folder = new Folder(); + var refreshed = 0; + + Task Refresh(JourneyRunOptions current, CancellationToken _) + { + refreshed++; + return Task.FromResult(current); + } + + var unmet = new JourneyRunOptions + { + Files = [folder.Add("a/one", "open /")], + Preconditions = [Build], + RefreshAfterPreconditions = Refresh, + GitHubActions = false + }; + var none = new JourneyRunOptions { Files = [folder.Add("a/two", "open /")], RefreshAfterPreconditions = Refresh, GitHubActions = false }; + + var first = await RunAsync(unmet, wait: _ => new PreconditionResult(false, "no", 1, TimeSpan.Zero)); + var second = await RunAsync(none); + + first.Exit.Should().Be(JourneyExitCode.Precondition); + second.Exit.Should().Be(JourneyExitCode.Ok); + refreshed.Should().Be(0); + } + + [Test] + public async Task APreconditionIsWaitedForBeforeThePlanEvenWhenEveryJourneyIsSkipped() + { + using var folder = new Folder(); + var options = new JourneyRunOptions + { + Files = [folder.Add("a/needs", "# @requires build-type:isolated\nopen /")], + BuildType = "assembler", + Preconditions = [Build], + AllowAllSkipped = true, + GitHubActions = false, + }; + + var (exit, output, error) = await RunAsync(options, wait: _ => new PreconditionResult(false, "not yet", 1, TimeSpan.Zero)); + + exit.Should().Be(JourneyExitCode.Precondition); + error.Should().StartWith("Error: not yet"); + output.Should().NotContain("skipped a/needs"); } [Test] diff --git a/tests/Nullean.Snapshots.Browser.Tests/JourneyWaitOptionTests.cs b/tests/Nullean.Snapshots.Browser.Tests/JourneyWaitOptionTests.cs deleted file mode 100644 index 6544ac9..0000000 --- a/tests/Nullean.Snapshots.Browser.Tests/JourneyWaitOptionTests.cs +++ /dev/null @@ -1,101 +0,0 @@ -using AwesomeAssertions; -using Nullean.Cdp; -using Nullean.Snapshots.Cli; - -namespace Nullean.Snapshots.Browser.Tests; - -/// How --wait-for and its options become preconditions. -public class JourneyWaitOptionTests -{ - private static (bool Ok, List Preconditions, string Problem) Parse(JourneyRunArgs args) - { - var ok = JourneyCli.TryParseWaits(args, out var preconditions, out var problem); - return (ok, preconditions, problem); - } - - [Test] - public void AHeaderAnAttributeAndTheTextOfAnElementAreRead() - { - var (ok, preconditions, _) = Parse(new JourneyRunArgs - { - WaitFor = - [ - "header:x-docs-build==1.74.1", - "meta[name='docs:build']@content==1.74.1", - "h1#build==Ready", - "a[href*='x']@href=*/docs", - @"header:Server=~^Amazon" - ], - }); - - ok.Should().BeTrue(); - preconditions - .Select(p => (p.Header, p.Selector, p.Attribute, p.Expected, p.Match)) - .Should() - .Equal( - ("x-docs-build", null, null, "1.74.1", TextMatch.Exact), - (null, "meta[name='docs:build']", "content", "1.74.1", TextMatch.Exact), - (null, "h1#build", null, "Ready", TextMatch.Exact), - (null, "a[href*='x']", "href", "/docs", TextMatch.Contains), - ("Server", null, null, "^Amazon", TextMatch.Regex) - ); - } - - [Test] - public void TheOptionsApplyToEveryWaitWithTheirDefaults() - { - var defaults = Parse(new JourneyRunArgs { WaitFor = ["header:x==1"] }).Preconditions.Single(); - var given = Parse(new JourneyRunArgs - { - WaitFor = ["header:x==1", "header:y==2"], - WaitPath = "/docs", - WaitTimeout = "90s", - WaitInterval = "250ms", - WaitRetries = 0 - }).Preconditions; - - (defaults.Path, defaults.Timeout, defaults.Interval, defaults.Retries).Should().Be(("./", TimeSpan.FromSeconds(10), TimeSpan.FromMilliseconds(500), 2)); - given.Should().OnlyContain( - p => p.Path == "/docs" && p.Timeout == TimeSpan.FromSeconds(90) && p.Interval == TimeSpan.FromMilliseconds(250) && p.Retries == 0 - ); - } - - [Test] - [Arguments("2m", 120.0)] - [Arguments("1500ms", 1.5)] - [Arguments("7", 7.0)] - [Arguments("0.5s", 0.5)] - public void DurationsTakeAUnitOrBareSeconds(string value, double seconds) - { - JourneyCli.TryParseDuration("--wait-timeout", value, TimeSpan.Zero, out var duration, out _).Should().BeTrue(); - - duration.TotalSeconds.Should().Be(seconds); - } - - [Test] - [Arguments("nonsense")] - [Arguments("header:x")] - [Arguments("==1")] - [Arguments("header:==1")] - [Arguments("h1@==1")] - [Arguments("h1==")] - public void AWaitThatIsNotOneIsAUsageError(string value) - { - var (ok, _, problem) = Parse(new JourneyRunArgs { WaitFor = [value] }); - - ok.Should().BeFalse(); - problem.Should().StartWith("Error: --wait-for '" + value + "'"); - } - - [Test] - public void ABadDurationOrRetriesOrAnOptionWithoutAWaitIsAUsageError() - { - Parse(new JourneyRunArgs { WaitFor = ["header:x==1"], WaitTimeout = "soon" }) - .Problem - .Should() - .Be("Error: --wait-timeout 'soon' is not a duration: use 10s, 500ms or 2m."); - Parse(new JourneyRunArgs { WaitFor = ["header:x==1"], WaitRetries = -1 }).Problem.Should().Be("Error: --wait-retries -1 is not zero or more."); - Parse(new JourneyRunArgs { WaitTimeout = "5s" }).Problem.Should().StartWith("Error: --wait-path, --wait-timeout"); - Parse(new JourneyRunArgs()).Ok.Should().BeTrue(); - } -} diff --git a/tests/Nullean.Snapshots.Browser.Tests/JourneyWaitTests.cs b/tests/Nullean.Snapshots.Browser.Tests/JourneyWaitTests.cs new file mode 100644 index 0000000..5af7c1d --- /dev/null +++ b/tests/Nullean.Snapshots.Browser.Tests/JourneyWaitTests.cs @@ -0,0 +1,136 @@ +using AwesomeAssertions; +using Nullean.Cdp; +using Nullean.Cdp.Journeys; +using Nullean.Snapshots.Browser.Console; +using Nullean.Snapshots.Cli; + +namespace Nullean.Snapshots.Browser.Tests; + +/// How the wait flags of a command line become preconditions. +public class JourneyWaitTests +{ + private static JourneyWaitParse Parse(params string[] checks) => JourneyWait.Parse(new JourneyWaitFlags { Checks = checks }); + + private static (string? Header, string? Selector, string? Attribute, string Expected, TextMatch Match) Shape(JourneyPrecondition p) => + (p.Header, p.Selector, p.Attribute, p.Expected, p.Match); + + [Test] + public void AHeaderAnAttributeAndTheTextOfAnElementAreRead() + { + var parsed = Parse( + "header:x-docs-build==1.74.1", + "meta[name='docs:build']@content==1.74.1", + "h1#build==Ready", + "a[href*='x']@href=*/docs", + @"header:Server=~^Amazon" + ); + + parsed.Valid.Should().BeTrue(); + parsed + .Preconditions + .Select(Shape) + .Should() + .Equal( + ("x-docs-build", null, null, "1.74.1", TextMatch.Exact), + (null, "meta[name='docs:build']", "content", "1.74.1", TextMatch.Exact), + (null, "h1#build", null, "Ready", TextMatch.Exact), + (null, "a[href*='x']", "href", "/docs", TextMatch.Contains), + ("Server", null, null, "^Amazon", TextMatch.Regex) + ); + } + + [Test] + public void AnAtSignOrAnOperatorInsideQuotesBelongsToTheSelector() + { + var parsed = Parse( + "a[href='mailto:docs@elastic.co']==Contact", + "a[href='mailto:docs@elastic.co']@href==mailto:docs@elastic.co", + "a[title=\"a==b\"]==Link", + "a[title='x=~y'][href]@title=*y", + "p@lang==en@gb" + ); + + parsed.Valid.Should().BeTrue(parsed.Problem); + parsed + .Preconditions + .Select(Shape) + .Should() + .Equal( + (null, "a[href='mailto:docs@elastic.co']", null, "Contact", TextMatch.Exact), + (null, "a[href='mailto:docs@elastic.co']", "href", "mailto:docs@elastic.co", TextMatch.Exact), + (null, "a[title=\"a==b\"]", null, "Link", TextMatch.Exact), + (null, "a[title='x=~y'][href]", "title", "y", TextMatch.Contains), + (null, "p", "lang", "en@gb", TextMatch.Exact) + ); + } + + [Test] + public void TheOptionsApplyToEveryCheckWithTheirDefaults() + { + var defaults = Parse("header:x==1").Preconditions.Single(); + var given = JourneyWait.Parse(new JourneyWaitFlags + { + Checks = ["header:x==1", "header:y==2"], + Path = "/docs", + Timeout = "90s", + Interval = "250ms", + Retries = 0 + }).Preconditions; + + (defaults.Path, defaults.Timeout, defaults.Interval, defaults.Retries).Should().Be(("./", TimeSpan.FromSeconds(10), TimeSpan.FromMilliseconds(500), 2)); + given.Should().OnlyContain( + p => p.Path == "/docs" && p.Timeout == TimeSpan.FromSeconds(90) && p.Interval == TimeSpan.FromMilliseconds(250) && p.Retries == 0 + ); + } + + [Test] + [Arguments("2m", 120.0)] + [Arguments("1500ms", 1.5)] + [Arguments("7", 7.0)] + [Arguments("0.5s", 0.5)] + public void DurationsTakeAUnitOrBareSeconds(string value, double seconds) + { + var parsed = JourneyWait.Parse(new JourneyWaitFlags { Checks = ["header:x==1"], Timeout = value }); + + parsed.Preconditions.Single().Timeout.TotalSeconds.Should().Be(seconds); + } + + [Test] + [Arguments("nonsense")] + [Arguments("header:x")] + [Arguments("==1")] + [Arguments("header:==1")] + [Arguments("h1@==1")] + [Arguments("h1==")] + [Arguments("a[href='x==1']")] + public void ACheckThatIsNotOneIsAProblem(string value) + { + var parsed = Parse(value); + + parsed.Valid.Should().BeFalse(); + parsed.Preconditions.Should().BeEmpty(); + parsed.Problem.Should().StartWith("--wait-for '" + value + "'"); + } + + [Test] + public void ABadDurationOrRetriesOrOptionsWithoutACheckAreProblems() + { + JourneyWait + .Parse(new JourneyWaitFlags { Checks = ["header:x==1"], Timeout = "soon" }) + .Problem + .Should() + .Be("--wait-timeout 'soon' is not a duration: use 10s, 500ms or 2m."); + JourneyWait.Parse(new JourneyWaitFlags { Checks = ["header:x==1"], Retries = -1 }).Problem.Should().Be("--wait-retries -1 is not zero or more."); + JourneyWait.Parse(new JourneyWaitFlags { Timeout = "5s" }).Problem.Should().StartWith("--wait-path, --wait-timeout"); + JourneyWait.Parse(new JourneyWaitFlags()).Should().Be(new JourneyWaitParse([])); + } + + [Test] + public void TheCliPrefixesAProblemAsAUsageError() + { + JourneyCli.TryParseWaits(new JourneyRunArgs { WaitFor = ["nonsense"] }, out var preconditions, out var problem).Should().BeFalse(); + + preconditions.Should().BeEmpty(); + problem.Should().StartWith("Error: --wait-for 'nonsense' is not a check"); + } +} diff --git a/tests/Nullean.Snapshots.PublicApi.Tests/.snapshots/PackageApiTests/Keeps_its_public_api.Nullean.Snapshots.Browser.Console.0aef5a5d.verified.api b/tests/Nullean.Snapshots.PublicApi.Tests/.snapshots/PackageApiTests/Keeps_its_public_api.Nullean.Snapshots.Browser.Console.0aef5a5d.verified.api index 1572722..bdbf84c 100644 --- a/tests/Nullean.Snapshots.PublicApi.Tests/.snapshots/PackageApiTests/Keeps_its_public_api.Nullean.Snapshots.Browser.Console.0aef5a5d.verified.api +++ b/tests/Nullean.Snapshots.PublicApi.Tests/.snapshots/PackageApiTests/Keeps_its_public_api.Nullean.Snapshots.Browser.Console.0aef5a5d.verified.api @@ -120,6 +120,7 @@ namespace Nullean.Snapshots.Browser.Console int ParseErrorExitCode { get; init; } IReadOnlyList Preconditions { get; init; } bool ReadOnly { get; init; } + Func>? RefreshAfterPreconditions { get; init; } string? Rerun { get; init; } int Retries { get; init; } bool ScreenshotOnFailure { get; init; } @@ -133,6 +134,38 @@ namespace Nullean.Snapshots.Browser.Console JourneyViewport? Viewport { get; init; } IReadOnlyList WritableUrls { get; init; } + static class JourneyWait + static JourneyWaitParse Parse(JourneyWaitFlags flags) + + sealed record JourneyWaitFlags : IEquatable + JourneyWaitFlags() + IReadOnlyList Checks { get; init; } + string? Interval { get; init; } + string? Path { get; init; } + int? Retries { get; init; } + string? Timeout { get; init; } + JourneyWaitFlags $() + bool Equals(JourneyWaitFlags? other) + override bool Equals(object? obj) + override int GetHashCode() + override string ToString() + static bool operator ==(JourneyWaitFlags? left, JourneyWaitFlags? right) + static bool operator !=(JourneyWaitFlags? left, JourneyWaitFlags? right) + + sealed record JourneyWaitParse : IEquatable + JourneyWaitParse(IReadOnlyList Preconditions, string? Problem = null) + IReadOnlyList Preconditions { get; init; } + string? Problem { get; init; } + bool Valid { get; } + JourneyWaitParse $() + void Deconstruct(out IReadOnlyList Preconditions, out string? Problem) + bool Equals(JourneyWaitParse? other) + override bool Equals(object? obj) + override int GetHashCode() + override string ToString() + static bool operator ==(JourneyWaitParse? left, JourneyWaitParse? right) + static bool operator !=(JourneyWaitParse? left, JourneyWaitParse? right) + static class TerminalPicture const int ZoomColumns = 60 const int ZoomRows = 12