Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -706,9 +706,16 @@ image or a transcript is not something to read in a source file.
- **Its own exit code, `Precondition` (4).** It is neither a failed journey (1) nor a usage error (2), so a workflow can tell "the deploy
is not live" from "the site is wrong". Nothing runs, `--retries` does not apply, and the run still reports: the reason on
stderr, a failed `preconditions/wait` case in JUnit and the step summary, and an error annotation on GitHub Actions.
- **Not applied when every journey is skipped**, like Chrome itself: a run that checks nothing needs no site.
- **They come before the journeys are planned.** A deploy that has just landed can change the flags and the build type that `@requires`
and `feature` blocks are judged against, so `JourneyRunOptions.RefreshAfterPreconditions` lets the caller read them again once the
wait is met, and the plan, the skips and the run use the refreshed options. The wait therefore also happens when every journey
would be skipped; Chrome starts for it.
- **The flags are parsed here, once.** `JourneyWait.Parse` (`JourneyWaitFlags` in, `JourneyWaitParse` out) is what `snap journey run`
and any tool on `JourneyConsoleRunner` call, so the syntax cannot drift between them. `==`, `=~` and `=*` are found outside quotes
and brackets, and the last `@` outside them starts the attribute: `a[href='mailto:[email protected]']==Contact` is a selector
with no attribute. A problem is plain text naming the flag; the caller adds its own prefix.
- **`--wait-for` is `target==value`, not `target=value`.** A CSS selector may hold `=`, `~=` and `*=` (`meta[name='docs:build']`,
`a[href*='x']`), but never `==`, `=~` or `=*`, so those three are the operators and nothing needs escaping. `header:` marks a
`a[href*='x']`), and quoted text may hold anything, so `==`, `=~` and `=*` outside quotes and brackets are the operators. `header:` marks a
response header, `@attribute` an attribute, and no `@` means the element's text.
- **`compare` waits for the target only.** The baseline is another release, so the value that proves the target deployed is
not the baseline's.
Expand Down
2 changes: 1 addition & 1 deletion docs/formats/browser.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ are files, not part of the transcript: pixels differ between machines.
| `ReadOnly`, `WritableUrls` | `--read-only`, `--writable` | refuse writes, except to these URL globs |
| `Viewport` | `--viewport` | the size every journey starts with; Chrome's default is a phone layout on a responsive site |
| `UserAgent`, `Headers` | `--user-agent`, `--header` | what every page of the run sends, such as the identity a site uses to tell a monitor from a visitor |
| `Preconditions` | `--wait-for`, `--wait-path`, `--wait-timeout`, `--wait-interval`, `--wait-retries` | wait for a response header or an element value before any journey runs; unmet ends the run with exit code 4 (`JourneyExitCode.Precondition`) |
| `Preconditions` | `--wait-for`, `--wait-path`, `--wait-timeout`, `--wait-interval`, `--wait-retries` | wait for a response header or an element value before any journey runs; unmet ends the run with exit code 4 (`JourneyExitCode.Precondition`). A tool parses the flags with `JourneyWait.Parse` and can re-read the site's flags once the wait is met with `RefreshAfterPreconditions` |
| `Retries` | `--retries` | run a journey that fails again this many times; one that passes on a retry passes the run and is reported as flaky (`JourneyOutcome.Flaky`, `EarlierAttempts`) |
| `ScreenshotOnFailure` | `--screenshot-on-failure` | take a screenshot of the viewport when a step fails (`JourneyFailure.Screenshot`), written to `failure.png` in the artifacts folder |
| `OnStep` | | called before each step, for a progress display |
Expand Down
2 changes: 1 addition & 1 deletion docs/workflow/snap.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ with exit code 4 and the reason, instead of failing every journey. A check is on
--wait-for "header:Server=~^Amazon" # =~ matches a regular expression, =* contains
```

The operators `==`, `=~` and `=*` never occur in a CSS selector, so a selector needs no escaping. `--wait-path` is the page
The operators `==`, `=~` and `=*` are read outside quotes and brackets, and the last `@` outside them starts the attribute, so a selector such as `a[href='mailto:[email protected]']` needs no escaping. `--wait-path` is the page
the checks look at, resolved against `--base-url` (default `./`). `--wait-timeout` (default `10s`; also `500ms`, `2m`) bounds the
wait in all and `--wait-interval` (`500ms`) spaces the looks. `--wait-retries` (default 2) is how many looks that fail, because the
page did not load or the connection dropped, are tried again before the wait gives up; a look that works and shows another
Expand Down
46 changes: 33 additions & 13 deletions src/Nullean.Snapshots.Browser.Console/JourneyConsoleRunner.cs
Original file line number Diff line number Diff line change
Expand Up @@ -54,29 +54,34 @@ CancellationToken cancellationToken
)
{
var clock = Stopwatch.StartNew();
var plan = options.Files.Select(f => Plan(f, options)).ToList();
reporter.Planned([.. plan.Where(p => p.Skip is not null).Select(p => (p.File, p.Skip!))], plan.Count(p => p.Journey is not null && p.Skip is null));

if (plan.Any(p => p.Journey is not null && p.Skip is null))
var browserStarted = false;
// What the site must show comes first, and before the journeys are planned: a deploy that has just landed can change the
// flags and build type they are judged against, so the caller gets to read them again once the site is as expected.
if (options.Preconditions.Count > 0)
{
try
{
await executor.EnsureReadyAsync(cancellationToken).ConfigureAwait(false);
}
catch (BrowserLaunchException e)
{
await error.WriteLineAsync(("Error: " + e.Message).AsMemory(), CancellationToken.None).ConfigureAwait(false);
if (!await StartBrowserAsync(executor, error, cancellationToken).ConfigureAwait(false))
return JourneyExitCode.Browser;
}

browserStarted = true;
if (await UnmetPreconditionAsync(options, executor, reporter, cancellationToken).ConfigureAwait(false) is { } unmet)
{
await error.WriteLineAsync(("Error: " + unmet).AsMemory(), CancellationToken.None).ConfigureAwait(false);
WritePreconditionReports(options, output, unmet, clock.Elapsed);
return JourneyExitCode.Precondition;
}

if (options.RefreshAfterPreconditions is { } refresh)
options = await refresh(options, cancellationToken).ConfigureAwait(false);
}

var plan = options.Files.Select(f => Plan(f, options)).ToList();
reporter.Planned([.. plan.Where(p => p.Skip is not null).Select(p => (p.File, p.Skip!))], plan.Count(p => p.Journey is not null && p.Skip is null));

if (
!browserStarted && plan.Any(p => p.Journey is not null && p.Skip is null) &&
!await StartBrowserAsync(executor, error, cancellationToken).ConfigureAwait(false)
)
return JourneyExitCode.Browser;

using var gate = new SemaphoreSlim(Math.Max(1, options.Parallel));
var launched = new LaunchFailure();
var tasks = plan.Select(item => RunOneAsync(item, options, executor, reporter, gate, launched, cancellationToken));
Expand Down Expand Up @@ -104,6 +109,21 @@ CancellationToken cancellationToken
return ExitCodeOf(outcomes, options);
}

// Starts Chrome; false after writing why when it cannot start.
private static async Task<bool> StartBrowserAsync(IJourneyExecutor executor, TextWriter error, CancellationToken cancellationToken)
{
try
{
await executor.EnsureReadyAsync(cancellationToken).ConfigureAwait(false);
return true;
}
catch (BrowserLaunchException e)
{
await error.WriteLineAsync(("Error: " + e.Message).AsMemory(), CancellationToken.None).ConfigureAwait(false);
return false;
}
}

// The message of the first precondition that was not met; null when all were.
private static async Task<string?> UnmetPreconditionAsync(
JourneyRunOptions options,
Expand Down
13 changes: 10 additions & 3 deletions src/Nullean.Snapshots.Browser.Console/JourneyRun.cs
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,19 @@ public sealed class JourneyRunOptions

/// <summary>
/// What the site must show before any journey runs, such as the build a deploy just shipped: waited for one after the other, in
/// the shared Chrome, each within its own timeout and retries. When one is not met nothing runs: the run writes the reason and
/// ends with <see cref="JourneyExitCode.Precondition"/>, and <see cref="JourneyRunOptions.JUnitPath"/> gets a failed
/// <c>preconditions/wait</c> case. Not applied when every journey is skipped, and never retried by <see cref="Retries"/>.
/// the shared Chrome, each within its own timeout and retries. They come first, before the journeys are planned. When one is not
/// met nothing runs: the run writes the reason and ends with <see cref="JourneyExitCode.Precondition"/>, and
/// <see cref="JourneyRunOptions.JUnitPath"/> gets a failed <c>preconditions/wait</c> case. Never retried by <see cref="Retries"/>.
/// </summary>
public IReadOnlyList<JourneyPrecondition> Preconditions { get; init; } = [];

/// <summary>
/// Called once every precondition is met and before the journeys are planned, with these options, to return the options the run
/// uses: a deploy that has just landed can change the flags and build type that <c>@requires</c> and <c>feature</c> blocks are
/// judged against, so a caller that read them before the wait reads them again here. Not called without preconditions.
/// </summary>
public Func<JourneyRunOptions, CancellationToken, Task<JourneyRunOptions>>? RefreshAfterPreconditions { get; init; }

/// <summary>The enabled feature flags of the site under test.</summary>
public IReadOnlySet<string> Features { get; init; } = new HashSet<string>(StringComparer.Ordinal);

Expand Down
203 changes: 203 additions & 0 deletions src/Nullean.Snapshots.Browser.Console/JourneyWait.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
using System.Globalization;
using Nullean.Cdp;
using Nullean.Cdp.Journeys;

namespace Nullean.Snapshots.Browser.Console;

/// <summary>
/// The command-line form of <see cref="JourneyRunOptions.Preconditions"/>, as the flags of <c>snap journey run</c> and any tool built on
/// <see cref="JourneyConsoleRunner"/> take them. Raw text, as typed: <see cref="JourneyWait.Parse"/> checks it.
/// </summary>
public sealed record JourneyWaitFlags
{
/// <summary>
/// The checks, each <c>header:Name==value</c> (a response header), <c>selector==value</c> (an element's text) or
/// <c>selector@attribute==value</c> (an attribute). <c>=~</c> in place of <c>==</c> matches a regular expression and <c>=*</c> contains.
/// </summary>
public IReadOnlyList<string> Checks { get; init; } = [];

/// <summary>The page the checks look at, resolved against the base URL. Default: <c>./</c>.</summary>
public string? Path { get; init; }

/// <summary>How long the wait lasts in all: <c>10s</c>, <c>500ms</c>, <c>2m</c>, or a number of seconds. Default: 10 seconds.</summary>
public string? Timeout { get; init; }

/// <summary>How long to wait between looks, in the same form. Default: 500 milliseconds.</summary>
public string? Interval { get; init; }

/// <summary>How many looks that fail are tried again. Default: <see cref="JourneyPrecondition.Retries"/>.</summary>
public int? Retries { get; init; }
}

/// <summary>What <see cref="JourneyWait.Parse"/> made of the flags: the preconditions, or what is wrong with them.</summary>
/// <param name="Preconditions">One per check; empty when there is a <paramref name="Problem"/> or when no check was given.</param>
/// <param name="Problem">Plain text naming the flag at fault, without a prefix; <c>null</c> when the flags are valid.</param>
public sealed record JourneyWaitParse(IReadOnlyList<JourneyPrecondition> Preconditions, string? Problem = null)
{
/// <summary>The flags are valid.</summary>
public bool Valid => Problem is null;
}

/// <summary>Reads the wait flags of a command line into <see cref="JourneyPrecondition"/>s.</summary>
public static class JourneyWait
{
private static readonly (string Token, TextMatch Match)[] Operators = [("==", TextMatch.Exact), ("=~", TextMatch.Regex), ("=*", TextMatch.Contains)];

/// <summary>
/// Reads <paramref name="flags"/>. The operators <c>==</c>, <c>=~</c> and <c>=*</c> are found outside quotes and brackets, and the
/// last <c>@</c> outside them starts the attribute, so a selector such as <c>meta[name='docs:build']</c> or
/// <c>a[href='mailto:[email protected]']</c> needs no escaping. Timeout, interval and retries apply to every check.
/// </summary>
public static JourneyWaitParse Parse(JourneyWaitFlags flags)
{
ArgumentNullException.ThrowIfNull(flags);
if (flags.Checks.Count == 0)
return flags.Path is not null || flags.Timeout is not null || flags.Interval is not null || flags.Retries is not null
? new JourneyWaitParse([], "--wait-path, --wait-timeout, --wait-interval and --wait-retries describe a --wait-for; give one.")
: new JourneyWaitParse([]);

if (
!TryDuration("--wait-timeout", flags.Timeout, TimeSpan.FromSeconds(10), out var timeout, out var problem) ||
!TryDuration("--wait-interval", flags.Interval, TimeSpan.FromMilliseconds(500), out var interval, out problem)
)
return new JourneyWaitParse([], problem);
if (flags.Retries is < 0)
return new JourneyWaitParse([], $"--wait-retries {flags.Retries} is not zero or more.");

var preconditions = new List<JourneyPrecondition>();
foreach (var check in flags.Checks)
{
if (!TryCheck(check, out var parsed, out problem))
return new JourneyWaitParse([], problem);
preconditions.Add(parsed with { Path = flags.Path ?? "./", Timeout = timeout, Interval = interval, Retries = flags.Retries ?? parsed.Retries });
}

return new JourneyWaitParse(preconditions);
}

private static bool TryCheck(string value, out JourneyPrecondition precondition, out string problem)
{
problem = string.Empty;
precondition = null!;
var (index, length, match) = FindOperator(value);
var target = index < 0 ? string.Empty : value[..index];
var expected = index < 0 ? string.Empty : value[(index + length)..];
if (index < 0 || target.Length == 0 || expected.Length == 0)
{
problem =
$"--wait-for '{value}' is not a check: use header:Name==value, selector==value or selector@attribute==value ('=~' matches a regular expression, '=*' contains).";
return false;
}

if (target.StartsWith("header:", StringComparison.OrdinalIgnoreCase))
{
var name = target["header:".Length..];
if (name.Length == 0)
{
problem = $"--wait-for '{value}' names no header.";
return false;
}

precondition = new JourneyPrecondition { Path = "./", Header = name, Expected = expected, Match = match };
return true;
}

string? attribute = null;
if (LastTopLevel(target, '@') is var at and >= 0)
{
attribute = target[(at + 1)..];
target = target[..at];
}

if (target.Length == 0 || attribute is { Length: 0 })
{
problem = $"--wait-for '{value}' is not a check: the selector or the attribute is empty.";
return false;
}

precondition = new JourneyPrecondition { Path = "./", Selector = target, Attribute = attribute, Expected = expected, Match = match };
return true;
}

// The first operator outside quotes and brackets, which is where the target ends and the expected value starts.
private static (int Index, int Length, TextMatch Match) FindOperator(string value)
{
var scan = new Scan();
for (var i = 0; i < value.Length; i++)
{
if (scan.Outside(value[i]))
{
foreach (var (token, match) in Operators)
if (string.CompareOrdinal(value, i, token, 0, token.Length) == 0)
return (i, token.Length, match);
}
}

return (-1, 0, TextMatch.Exact);
}

private static int LastTopLevel(string text, char wanted)
{
var scan = new Scan();
var last = -1;
for (var i = 0; i < text.Length; i++)
{
if (scan.Outside(text[i]) && text[i] == wanted)
last = i;
}

return last;
}

// Tracks whether the scan is inside a quoted string or a [bracket], as in a CSS attribute selector.
private struct Scan
{
private char _quote;
private int _depth;

// Whether this character sits outside quotes and brackets; updates the state for the next one.
public bool Outside(char c)
{
if (_quote != 0)
{
if (c == _quote)
_quote = '\0';
return false;
}

switch (c)
{
case '\'' or '"':
_quote = c;
return false;
case '[':
_depth++;
return false;
case ']' when _depth > 0:
_depth--;
return false;
default:
return _depth == 0;
}
}
}

private static bool TryDuration(string flag, string? value, TimeSpan fallback, out TimeSpan duration, out string problem)
{
problem = string.Empty;
duration = fallback;
if (value is null)
return true;
var (digits, unit) = value.EndsWith("ms", StringComparison.Ordinal)
? (value[..^2], 0.001)
: value.EndsWith('s') ? (value[..^1], 1.0) : value.EndsWith('m') ? (value[..^1], 60.0) : (value, 1.0);
if (double.TryParse(digits, NumberStyles.Float, CultureInfo.InvariantCulture, out var number) && number > 0)
{
duration = TimeSpan.FromSeconds(number * unit);
return true;
}

problem = $"{flag} '{value}' is not a duration: use 10s, 500ms or 2m.";
return false;
}
}
Loading
Loading