Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/ARIADNE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1236,6 +1236,18 @@ A capture is made repeatable by:
The test then renders one more frame and requires it to be identical, so a UI that is still
changing fails as unsettled rather than as a flaky mismatch.

To tell the UI from the rasterizer, each of the two frames is fingerprinted with what ImGui
handed GL: its draw lists (vertices, indices, commands, clip rects, texture ids) and every
texture's size and pixels. The settle line says whether the UI was the same both times, and
names the first differing pixel.

macOS CI's Apple Software Renderer sometimes draws the same UI two ways in a process's first
frames. It has only ever hit the first document. In 2026-10 it moved 8 pixels by 1 between
frames whose fingerprints were identical. So when the UI is identical and only pixels differ,
the later frame becomes the capture and the test renders another, up to 10 times, until two
frames agree exactly. The log says how many extra frames that took. A UI that changed fails at
once, as does a rasterizer that never settles.

**Comparing.** A pixel mismatches when a channel differs by more than 40. A document passes
while at most 0.002% of its pixels (13) mismatch. The goldens are made under Mesa llvmpipe, the
rasterizer of the Linux CI job (Xvfb + cvcpkg's `mesa`). These limits were measured:
Expand Down
138 changes: 129 additions & 9 deletions src/cvcGL/test/cvcgl_ari_golden.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
// least 700 px on its short side, so the small-screen rule never closes them), ImGui's frame delta
// pinned, and several frames at a fixed dt before the capture, so new windows have finished
// appearing. The test then renders one frame more and requires it to be identical to the capture.
// When ImGui drew exactly the same UI in both and only the rasterizer moved (the Apple Software
// Renderer, in a process's first frames), it renders up to kRasterSettleFrames more until two
// agree.
//
// Goldens are made under Mesa llvmpipe, the rasterizer of the Linux and Windows CI jobs. Other
// rasterizers, and other Mesa versions, draw anti-aliased edges and shading a little differently,
Expand All @@ -28,6 +31,7 @@
#include "streaming_test_util.h"

#include <algorithm>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
Expand Down Expand Up @@ -55,6 +59,7 @@ namespace {
constexpr int kWidth = 960, kHeight = 720; // short side >= 700: the overlay keeps its panels open
constexpr double kDt = 1.0 / 30.0; // the host frame step AND ImGui's frame delta
constexpr int kMinFrames = 4; // frames before the capture (ImGui auto-fit needs 2)
constexpr int kRasterSettleFrames = 10; // more frames for a rasterizer drawing one UI two ways
const double kBackground[3] = {0.09, 0.10, 0.12};

// One tolerance for every rasterizer, from these measurements against the goldens (Mesa 25.3.6
Expand Down Expand Up @@ -136,6 +141,79 @@ double drawnFraction(const Image &img) {
return double(n) / double(img.w * img.h);
}

// The first pixel two frames disagree on, with both colours ("" when they are identical).
std::string firstDifference(const Image &a, const Image &b) {
if (a.empty() || b.empty() || a.w != b.w || a.h != b.h)
return "";
for (int y = 0; y < a.h; ++y)
for (int x = 0; x < a.w; ++x) {
const unsigned char *p = a.at(x, y), *q = b.at(x, y);
if (p[0] != q[0] || p[1] != q[1] || p[2] != q[2]) {
char buf[96];
std::snprintf(buf, sizeof buf, "first at (%d,%d): %d,%d,%d then %d,%d,%d", x, y, p[0], p[1],
p[2], q[0], q[1], q[2]);
return buf;
}
}
return "";
}

// What ImGui handed the GL backend for the last rendered frame: its draw lists (vertices, indices,
// commands and their clip rects and textures) and every texture's size and pixels. Two frames with
// equal fingerprints asked GL to draw the same UI, so a pixel difference between them comes from
// the rasterizer, not from the UI.
struct DrawFingerprint {
bool valid = false; // false: no draw data (or no ImGui in this build), so nothing is known
uint64_t lists = 0, textures = 0;
int vertices = 0;
std::string textureSizes;
bool sameUi(const DrawFingerprint &o) const {
return valid && o.valid && lists == o.lists && textures == o.textures && vertices == o.vertices;
}
};

DrawFingerprint drawFingerprint(cvc::gl::ImGuiOverlay &ui) {
DrawFingerprint fp;
#ifndef CVC_ENABLE_IMGUI
(void)ui;
#else
ImGui::SetCurrentContext(ui.imguiContext());
const ImDrawData *dd = ImGui::GetDrawData();
if (!dd || !dd->Valid)
return fp;
auto fnv = [](uint64_t &h, const void *data, size_t n) {
const auto *b = static_cast<const unsigned char *>(data);
for (size_t i = 0; i < n; ++i)
h = (h ^ b[i]) * 1099511628211ull;
};
fp.lists = fp.textures = 1469598103934665603ull;
for (const ImDrawList *dl : dd->CmdLists) {
fnv(fp.lists, dl->VtxBuffer.Data, dl->VtxBuffer.Size * sizeof(ImDrawVert));
fnv(fp.lists, dl->IdxBuffer.Data, dl->IdxBuffer.Size * sizeof(ImDrawIdx));
for (const ImDrawCmd &c : dl->CmdBuffer) {
const ImTextureID tex = c.GetTexID();
fnv(fp.lists, &c.ClipRect, sizeof c.ClipRect);
fnv(fp.lists, &tex, sizeof tex);
fnv(fp.lists, &c.VtxOffset, sizeof c.VtxOffset);
fnv(fp.lists, &c.IdxOffset, sizeof c.IdxOffset);
fnv(fp.lists, &c.ElemCount, sizeof c.ElemCount);
}
fp.vertices += dl->VtxBuffer.Size;
}
if (dd->Textures)
for (const ImTextureData *t : *dd->Textures) {
fnv(fp.textures, &t->Width, sizeof t->Width);
fnv(fp.textures, &t->Height, sizeof t->Height);
if (t->Pixels)
fnv(fp.textures, t->Pixels, static_cast<size_t>(t->GetSizeInBytes()));
fp.textureSizes += (fp.textureSizes.empty() ? "" : " ") + std::to_string(t->Width) + "x" +
std::to_string(t->Height);
}
fp.valid = true;
#endif
return fp;
}

#ifdef CVC_ENABLE_IMGUI
void injectPointer(cvc::gl::ImGuiOverlay &ui, const GoldenCase *gc, int frame) {
if (!gc)
Expand Down Expand Up @@ -199,28 +277,70 @@ bool runDocument(const fs::path &doc, const fs::path &goldenDir, const fs::path
// no X display, which imgui's GLX-based loader cannot reach) nothing here can be tested.
if (!ui.enabled())
return false;
rt.frame(kDt);
const Image actual =
cvcgl_golden::fromFrameRGB(view.frameRGB(), view.frameWidth(), view.frameHeight());
rt.frame(kDt);
const Image next =
cvcgl_golden::fromFrameRGB(view.frameRGB(), view.frameWidth(), view.frameHeight());
// The capture, then one frame more, which must draw the same pixels. A UI that is still changing
// fails here. The one other way two frames differ is the rasterizer: ImGui handed GL exactly the
// same UI (the same fingerprint) and some pixels still moved. macOS CI's Apple Software Renderer
// does that, by 1 in a few pixels, in a process's first frames. Then the later frame becomes the
// capture and one more is drawn, up to kRasterSettleFrames times, and the log says so.
auto render = [&] {
rt.frame(kDt);
return cvcgl_golden::fromFrameRGB(view.frameRGB(), view.frameWidth(), view.frameHeight());
};
Image actual = render();
DrawFingerprint actualUi = drawFingerprint(ui);
Image next;
DrawFingerprint nextUi;
cvcgl_golden::Comparison settled;
std::string rasterNoise; // the first same-UI difference, when the rasterizer needed more frames
int rasterFrames = 0;
for (;;) {
next = render();
nextUi = drawFingerprint(ui);
settled = cvcgl_golden::compare(next, actual, {0, 0.0});
if (settled.pass || !actualUi.sameUi(nextUi) || rasterFrames == kRasterSettleFrames)
break;
if (rasterNoise.empty())
rasterNoise =
cvcgl_golden::describe(settled, {0, 0.0}) + "; " + firstDifference(actual, next);
++rasterFrames;
actual = std::move(next);
actualUi = nextUi;
}
for (const std::string &w : rt.take_warnings())
std::printf(" runtime: %s\n", w.c_str());

const std::string renderer = glRenderer(view);
const bool llvmpipe = renderer.find("llvmpipe") != std::string::npos;
std::printf(" %d frames at %dx%d on %s\n", frames + 2, kWidth, kHeight,
std::printf(" %d frames at %dx%d on %s\n", frames + 2 + rasterFrames, kWidth, kHeight,
renderer.empty() ? "(unknown renderer)" : renderer.c_str());

check(!actual.empty() && actual.w == kWidth && actual.h == kHeight,
name + ": captured a " + std::to_string(kWidth) + "x" + std::to_string(kHeight) + " frame");
const double drawn = drawnFraction(actual);
check(drawn > 0.01, name + ": the frame is more than the background",
std::to_string(100.0 * drawn) + "% of pixels drawn");
const cvcgl_golden::Comparison settled = cvcgl_golden::compare(next, actual, {0, 0.0});
std::string settledDetail = cvcgl_golden::describe(settled, {0, 0.0});
const std::string firstDiff = firstDifference(actual, next);
if (!firstDiff.empty())
settledDetail += "; " + firstDiff;
if (actualUi.sameUi(nextUi)) {
settledDetail += "; ImGui drew the same UI both times (" + std::to_string(nextUi.vertices) +
" vertices, textures " + nextUi.textureSizes + ")";
} else if (actualUi.valid && nextUi.valid) {
std::string what = actualUi.lists != nextUi.lists ? "draw lists" : "";
if (actualUi.textures != nextUi.textures)
what += what.empty() ? "textures" : " and textures";
settledDetail += "; ImGui's " + what + " changed between them (" +
std::to_string(actualUi.vertices) + " -> " + std::to_string(nextUi.vertices) +
" vertices, textures " + actualUi.textureSizes + " -> " + nextUi.textureSizes +
")";
}
if (rasterFrames > 0)
settledDetail += std::string("; with the same UI, the rasterizer ") +
(settled.pass ? "settled after " : "had not settled after ") +
std::to_string(rasterFrames) + " more frame(s) (first: " + rasterNoise + ")";
check(settled.pass, name + ": one more frame draws the same pixels (the UI has settled)",
cvcgl_golden::describe(settled, {0, 0.0}));
settledDetail);

// Which golden, and how strictly.
const std::string variant = goldenVariant();
Expand Down
Loading