Skip to content

TESTING.md

Latest commit

 

History

History
1447 lines (1108 loc) · 42.8 KB

TESTING.md

File metadata and controls

1447 lines (1108 loc) · 42.8 KB
Raw

Comprehensive Testing Guide for libcvc

Last Updated: December 29, 2025

Table of Contents

Overview

The libcvc library uses Google Test (gtest) for comprehensive unit testing. The test infrastructure is modern, well-documented, and integrated seamlessly with CMake. Tests are enabled by default and cover core functionality, multithreaded operations, async programming, CUDA GPU acceleration, and real-world scenarios.

Key Features:

  • 577 comprehensive tests with 100% pass rate
  • Automatic Google Test integration via CMake FetchContent
  • Code coverage analysis with lcov/gcov
  • Multithreaded concurrency testing
  • Futures API async programming tests
  • CUDA GPU acceleration tests
  • Real-world stress tests (SDF convergence, state_object patterns)

Test Suite Summary

Total Tests: 577 (100% Passing) ✅

Last Updated: April 2026

Component Tests Focus Areas
App 72 Application framework, data/property management, threading
State 158 State tree, hierarchies, signals, async operations, state_object pattern
Voxels 133 Volume data, algorithms, CUDA GPU paths
Volume 29 Spatial coordinates, interpolation, subvolumes
Geometry 119 Mesh operations, normals, I/O, SDF computation, isosurface extraction
Volume-Ops 42 Filtering, contrast enhancement, anisotropic diffusion, GDTV
HDF5 24 .cvc file format round-trip and metadata
Procedural Geometry typed fixtures Templated parameterized tests over generated meshes

Key Metrics:

  • Success Rate: 100%
  • Execution Time: ~280 seconds (includes intensive SDF convergence test)
  • Coverage: Target 64.6% overall, 94.1% on core components

Known Issue: StateTest.StateObjectStateName passes its assertions but encounters a segfault during test cleanup. This is a timing-related destructor issue that doesn't affect functionality.

Test Execution Speed

Test Suite Duration Notable Tests
app_test ~0.5s 114 tests, fast unit tests
state_test ~13s 105 tests, includes multithreaded + futures + async
voxels_test ~33s 130 tests, includes CUDA tests (22) with performance benchmarks
volume_test ~0.3s 29 tests, spatial operations
geometry_test ~0.4s 50 tests, Stanford Bunny I/O, deep copy
algorithm_test ~265s 6 tests, BunnyVolumeConvergence is ~228s

Building Tests

Prerequisites

Required:

  • CMake 3.15 or higher
  • C++20 compatible compiler (GCC 13+, Clang 17+, MSVC 19.29+); C++23 selectable via -DCMAKE_CXX_STANDARD=23
  • Boost libraries (>= 1.58): thread, date_time, regex, filesystem, system, chrono

Optional (for coverage):

  • lcov (for coverage report generation)
  • gcov (usually included with GCC)

CMake Option

Tests are controlled by the CVC_BUILD_TESTS CMake option, which is ON by default.

# Disable tests
cmake -DCVC_BUILD_TESTS=OFF /path/to/libcvc

# Enable tests (default)
cmake -DCVC_BUILD_TESTS=ON /path/to/libcvc

Building with Tests

# Configure with tests enabled (default)
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release

# Build the project including tests
cmake --build build -j$(nproc)

# Or build only specific test executables
cmake --build build --target app_test state_test

Google Test Integration

The tests use GoogleTest 1.17.0, resolved by CMake/SetupGTest.cmake. No manual installation of Google Test is required.

  1. An installed package, if there is one. find_package(GTest 1.17.0 CONFIG) searches CMAKE_PREFIX_PATH. The cvcpkg catalog's googletest is a build dependency of cvcpkg/recipes/libcvc, so a prefix made with cvcpkg install-deps cvcpkg/recipes/libcvc --prefix <p> already has it, and configuring against <p> needs no network.
  2. Otherwise, built from source. FetchContent downloads the same release tarball from GitHub (SHA-256 pinned) and builds it in the build tree.

The configure log names the one it used (-- GoogleTest 1.17.0: installed package (...) or ... building from source with FetchContent -- <reason>). Pass -DCMAKE_DISABLE_FIND_PACKAGE_GTest=ON to always build from source. With MSVC, an installed GoogleTest is used only when it matches the build's C runtime: a Release-only package is skipped for a Debug build, for instance, rather than mixing /MD and /MDd.

Running Tests

Using CTest (Recommended)

CTest is the standard way to run tests in CMake projects:

# Run all tests
cd build && ctest

# Run with verbose output on failure
ctest --output-on-failure

# Run tests in parallel
ctest -j$(nproc)

# Run specific test suite
ctest -R app_test
ctest -R state_test
ctest -R voxels_test

Using the Custom 'check' Target

A convenient check target is provided:

cmake --build build --target check

This runs all tests and displays output on failure.

Running Test Executables Directly

For more control and Google Test features:

# Run entire test suite
./build/bin/app_test
./build/bin/state_test
./build/bin/voxels_test
./build/bin/volume_test
./build/bin/geometry_test
./build/bin/algorithm_test

# Run with filters
./build/bin/app_test --gtest_filter=AppTest.DataSetAndGet
./build/bin/state_test --gtest_filter="*Futures*"
./build/bin/voxels_test --gtest_filter=VoxelsTest.BilateralFilter*

# List all tests
./build/bin/app_test --gtest_list_tests
./build/bin/state_test --gtest_list_tests

# Run tests multiple times (detect flakiness)
./build/bin/app_test --gtest_repeat=100

# Brief output (just pass/fail)
./build/bin/state_test --gtest_brief=1

Test Organization

Test Files

File Component Lines Tests
src/cvc/tests/app_test.cpp cvc::app ~1,400 114
src/cvc/tests/state_test.cpp cvc::state ~3,400 105
src/cvc/tests/voxels_test.cpp cvc::voxels ~3,520 130
src/cvc/tests/volume_test.cpp cvc::volume ~800 29
src/cvc/tests/geometry_test.cpp cvc::geometry ~900 50
src/cvc/tests/algorithm_test.cpp algorithms ~600 6
src/cvc/tests/CMakeLists.txt Build config ~60 -

Test Naming Convention

Tests follow the Google Test pattern:

TEST(TestSuiteName, TestName) {
  // Test implementation
}

Naming Guidelines:

  • Test suite name matches the class being tested (e.g., AppTest, StateTest)
  • Test name describes the specific behavior (e.g., DataSetAndGet, ConcurrentValueReads)
  • Use descriptive names that explain intent

Test Coverage by Component

App Component Tests (114 tests)

Focus: Application singleton, data/property management, threading, utilities

Click to expand App test categories

1. Singleton Pattern (1 test)

  • SingletonInstance

2. Data Management (13 tests)

  • DataSetAndGet (int, double, string, bool)
  • DataRemove
  • DataTypeInfo
  • DataTypeName
  • DataTypes (enumeration)

3. Data Type Registry (3 tests)

  • DataTypeRegistry
  • TypeRegistrationUniqueness
  • RegisterDataType

4. Property Management (12 tests)

  • PropertySetAndGet
  • PropertyRemove
  • PropertyMap
  • PropertyList operations (append, remove, unique)
  • PropertyListAppendUnique
  • MultipleProperties

5. Thread Management (8 tests)

  • ThreadKeyGeneration
  • ThreadKeyUniqueness
  • ThreadInfo
  • ThreadProgress
  • ThreadProgressMultiple
  • ThreadList

6. Mutex Management (3 tests)

  • MutexCreation
  • MutexReuse
  • MutexInfo

7. Utilities (3 tests)

  • Listify (vector ↔ string conversion)
  • Sleep function
  • ListifyRoundTrip

Coverage: 89.82% (441/491 lines in app.cpp)

State Component Tests (129 tests)

Focus: Hierarchical state system, signals, async futures, state_object pattern

Click to expand State test categories

1. Core State Management (10 tests)

  • SingletonInstance
  • Value storage (string, int, double, bool)
  • ValueCommaList, ValueConversion
  • ValueListUnique

2. Data Storage (4 tests)

  • DataSetAndGet
  • DataTypeInt, DataTypeDouble
  • Type-safe retrieval

3. Tree Structure (12 tests)

  • ChildCreation
  • FullName, ParentName
  • ChildrenListing, NumChildren
  • ChildrenWithRegex
  • DeepHierarchyNavigation
  • OperatorChaining
  • EmptyKeyHandling

4. Metadata (8 tests)

  • CommentSetAndGet
  • HiddenFlag
  • InitializedFlag
  • LastModification
  • Touch operations
  • InitializedFlagBehavior

5. Serialization (7 tests)

  • PropertyTreeConversion
  • PropertyTreeRoundTrip
  • JsonConversion, JsonRoundtrip
  • SaveAndRestore

6. Traversal (5 tests)

  • Traverse with callbacks
  • TraverseWithRegex
  • TraversalEnterExitSignals

7. Signals (3 tests)

  • ValueChangedSignal
  • DataChangedSignal
  • ChildChangedSignal

8. Multithreaded Tests (12 tests)

  • ConcurrentValueReads (10 threads, 1000 reads)
  • ConcurrentValueWrites (10 threads)
  • ConcurrentWritesToSameNode (20 threads)
  • ConcurrentDataOperations (8 threads)
  • ConcurrentSignalHandling
  • ConcurrentHierarchyCreation
  • ConcurrentTraversal
  • ConcurrentResetOperations
  • ConcurrentPropertyTreeOperations
  • DeadlockDetectionValueAndSignal
  • StateObjectMultithreaded (8 threads)
  • StressTestCombinedOperations (2,162 operations/sec)

9. Futures API Tests (11 tests)

  • ValueWithCallback
  • WaitForValue (blocking)
  • WaitForValueWithTimeout
  • ValueFutureGet
  • ValueFutureWaitFor
  • ValueFutureGetFor
  • DataWithCallback
  • WaitForData
  • WaitForDataWithTimeout
  • MultipleFuturesOnSameState
  • FutureProducerConsumerPattern

10. State Object Pattern Tests (11 tests)

  • StateObjectMultithreaded
  • StateObjectConfiguration
  • StateObjectDataProcessor
  • StateObjectRenderer
  • StateObjectAppSettings
  • StateObjectExternalAccess
  • StateObjectNestedPaths
  • StateObjectStateName
  • StateObjectMultipleInstances
  • StateObjectWithData
  • StateObjectAsyncHandlers

Coverage: 92.75% (243/262 lines in state.cpp)

Voxels Component Tests (130 tests)

Focus: Volume data structure, image processing algorithms, CUDA GPU acceleration

Click to expand Voxels test categories

1. Construction (5 tests)

  • Default, dimension-based, type-specified, copy, pointer-based

2. Voxel Access (5 tests)

  • Linear indexing, 3D coordinates, bounds checking

3. Dimension/Type Modification (4 tests)

  • Expansion, reduction, type conversion, precision

4. Min/Max Operations (9 tests)

  • Auto-calculation, manual setting, all data types, large volumes

5. Core Operations (7 tests)

  • Assignment, equality, fill, map, sub, resize

6. Advanced Operations (7 tests)

  • Histogram, copy-on-write, composite, raw data access

7. Copy Semantics (12 tests)

  • Shallow copy (default, shared data)
  • Deep copy (independent allocation)
  • Copy-on-write behavior
  • All data types, large volumes

8. Type Conversions (4 tests)

  • All 30 conversion paths (6 types × 5)
  • Precision loss handling
  • Negative value handling

9. Resize/Interpolation (4 tests)

  • Upsample, downsample, trilinear interpolation, non-uniform

10. Map Operations (4 tests)

  • Range expansion/shrinking, negative ranges, identity

11. Subvolume Extraction (3 tests)

  • Center, corner, single slice

12. Image Processing Algorithms (8 tests)

  • Bilateral filter (edge-preserving smoothing)
  • Contrast enhancement (resistor propagation)
  • Anisotropic diffusion (Perona-Malik model)
  • GDTV filter (gradient-domain total variation)

13. Composite Functions (3 tests)

  • Add, subtract, partial overlap handling

14. Edge Cases (4 tests)

  • Out-of-bounds exceptions, uninitialized states, large volumes

15. CUDA GPU Acceleration (22 tests)

  • Device management (availability, info, selection, enable/disable)
  • Memory operations (migration to/from GPU, multiple cycles, device switching)
  • Operations on GPU (modify, fill, map, subvolume)
  • Algorithms on GPU (bilateral filter, GDTV filter, min/max calculation)
  • Integration tests (copy with CUDA, different data types, large volumes)
  • Performance benchmarks (bilateral filter, GDTV filter with multiple configurations)

Coverage: 92.17% (306/332 lines in voxels.cpp)

Volume Component Tests (29 tests)

Focus: Spatial coordinate system, trilinear interpolation, subvolumes

  • Construction & Properties (8 tests)
  • Interpolation (7 tests)
  • Subvolumes (8 tests)
  • Operations (6 tests)

Coverage: 91.18% (124/136 lines in volume.cpp)

Geometry Component Tests (50 tests)

Focus: Triangle mesh operations, normals, quality improvement, deep copy semantics, I/O

Click to expand Geometry test categories

1. Basic Mesh Operations (8 tests)

  • Default construction
  • Copy construction and assignment
  • Triangle and vertex counting
  • Bounding box computation
  • Normal vector computation
  • Per-vertex normal averaging
  • Empty geometry handling
  • Extents calculation

2. Memory Semantics (3 tests) NEW

  • ShallowCopyDefault - Copy-on-write behavior (default)
  • DeepCopy - Independent memory allocation with copy(geom, true)
  • DeepCopyIndependence - Verify modifications don't affect original

Deep Copy Implementation:

// Shallow copy (default) - shares data via boost::shared_ptr
geometry g2;
g2.copy(g1);  // or g2.copy(g1, false)

// Deep copy - independent allocation
geometry g2;
g2.copy(g1, true);  // Creates new shared_ptr instances

When to Use Deep Copy:

  • Parallel processing: Multiple threads need independent meshes
  • Benchmarking: Avoid copy-on-write overhead in timing
  • Known modifications: Data will diverge immediately
  • Memory tracking: Explicit control over allocation timing

3. Mesh Quality & Smoothing (15 tests)

  • Laplacian smoothing (vertex movement validation)
  • Boundary-fixed smoothing
  • Topology preservation during smoothing
  • Triangle quality improvement (aspect ratio)
  • Multiple smoothing iterations
  • Surface projection after modification

4. Mesh Operations (10 tests)

  • Merge operations (index reindexing)
  • Triangle surface extraction
  • Wire interior generation
  • Normal inversion
  • Topology validation

5. File I/O (14 tests)

  • OFF format read/write
  • RAW format read/write
  • Geometry serialization
  • Stanford Bunny loading (real-world test: 34,835 vertices, 69,473 triangles)

Coverage: 79.00% (252/319 lines in geometry.cpp)

Algorithm Component Tests (6 tests)

Focus: High-level algorithms (SDF, isosurfacing, convergence)

  1. SDFBasic - Basic SDF computation
  2. IsoBasic - Basic isosurface extraction
  3. IsoWithDifferentIsovalues - Multiple isovalue testing
  4. SDFThenIsoRoundtrip - Pipeline validation
  5. BunnySDF_IsoRoundtrip - Real-world test (Stanford Bunny, ~5.5s)
  6. BunnyVolumeConvergence - Stress test (32³→256³, ~239s)

Code Coverage Metrics

Overall Project Coverage

Full Project (December 12, 2025):

  • Lines: 64.6% (10,272 of 15,903 lines)
  • Functions: 68.1% (6,848 of 10,056 functions)

Core Components Only:

  • Lines: 94.1% (429 of 456 lines in headers)
  • Functions: 91.2% (218/239 functions in headers)

Coverage by File

File Line Coverage Function Coverage Status
state.cpp 92.75% (243/262) 94.7% (355/375) ✅ Excellent
voxels.cpp 92.17% (306/332) 94.44% (17/18) ✅ Excellent
volume.cpp 91.18% (124/136) - ✅ Excellent
app.cpp 89.82% (441/491) - ✅ Good
geometry.cpp 79.00% (252/319) - ✅ Good

Header Files (Template instantiations):

  • geometry.h: 100%
  • volume.h: 83.3%
  • state.h: 64.0% (many template paths)
  • app.h: 57.6%
  • voxels.h: 26.5% (legacy code paths)

What's Covered

✅ Well Covered Areas:

  • Core state management (value/data operations)
  • Tree navigation and hierarchy
  • Multithreaded concurrent access
  • Futures API (blocking waits, callbacks, timeouts)
  • State object pattern (11 comprehensive tests)
  • Volume data structure (all 6 data types)
  • Image processing algorithms (bilateral, diffusion, GDTV, contrast)
  • CUDA GPU operations (20 dedicated tests)
  • Geometry mesh operations
  • SDF computation and convergence

⚠️ Areas Needing More Coverage:

  • Complex template instantiation paths
  • Error recovery scenarios
  • I/O edge cases (corrupt files, invalid formats)
  • Legacy compatibility code
  • Some specialized algorithms

Test Design Principles

1. Isolation

Each test is independent and cleans up after itself. The examples below use app as a cvc::app instance owned by the test (typically a fixture member or a file-scope test object):

TEST(StateTest, ValueSetAndGet) {
  cvc::app app;

  // Arrange
  state::instance(app)("test.value").value("test");
  
  // Act & Assert
  EXPECT_EQ(state::instance(app)("test.value").value(), "test");
  
  // Cleanup
  state::instance(app)("test").reset();
}

2. Namespace Usage

Tests use unique namespaces to avoid conflicts:

  • test.value.* - Value-related tests
  • test.data.* - Data storage tests
  • test.concurrent.* - Concurrency tests
  • test.future.* - Futures API tests

3. Thread Safety

The app and state classes are thread-safe. Tests include:

  • Single-threaded behavior validation
  • Concurrent access tests (12 tests)
  • Deadlock detection
  • Signal handling under load

4. Real-World Testing

Tests use realistic scenarios:

  • Stanford Bunny mesh (34,834 triangles)
  • Large volumes (256³ = 16.7M voxels)
  • Stress tests (thousands of operations)

Adding New Tests

Step 1: Choose the Appropriate File

  • app_test.cpp - For cvc::app functionality
  • state_test.cpp - For cvc::state functionality
  • voxels_test.cpp - For cvc::voxels functionality
  • volume_test.cpp - For cvc::volume functionality
  • geometry_test.cpp - For cvc::geometry functionality
  • Create new file for new components

Step 2: Follow the TEST Macro Pattern

TEST(TestSuiteName, TestName) {
  // Arrange: Set up test conditions
  cvc::volume vol(dimension(10, 10, 10));
  vol.fill(5.0);
  
  // Act: Perform the operation
  double value = vol(5, 5, 5);
  
  // Assert: Verify results
  EXPECT_DOUBLE_EQ(value, 5.0);
  
  // Cleanup (if needed)
  state::instance(app)("test").reset();
}

Step 3: Use Appropriate Assertions

Google Test provides various assertion macros:

  • EXPECT_EQ(a, b) - Values are equal
  • EXPECT_NE(a, b) - Values are not equal
  • EXPECT_TRUE(condition) - Condition is true
  • EXPECT_FALSE(condition) - Condition is false
  • EXPECT_LT/LE/GT/GE(a, b) - Comparison assertions
  • EXPECT_DOUBLE_EQ(a, b) - Floating point equality
  • EXPECT_THROW(statement, exception) - Exception testing
  • ASSERT_* versions - Stop test on failure

Step 4: Rebuild and Test

cmake --build build --target state_test
./build/bin/state_test --gtest_filter="*YourNewTest*"
ctest --output-on-failure

Common Testing Patterns

Testing Exception Handling

TEST(StateTest, TypeConversionError) {
  state::instance(app)("test").value(42);
  EXPECT_THROW({
    state::instance(app)("test").data<std::vector<int>>();
  }, cvc::type_conversion_error);
}

Testing with Fixtures

For tests requiring common setup:

class VoxelsTestFixture : public ::testing::Test {
protected:
  void SetUp() override {
    v = cvc::voxels(cvc::dimension(10, 10, 10), cvc::Float);
    v.fill(0.0);
  }
  
  void TearDown() override {
    // Cleanup if needed
  }
  
  cvc::voxels v;
};

TEST_F(VoxelsTestFixture, FillOperation) {
  v.fill(42.0);
  EXPECT_DOUBLE_EQ(v(5, 5, 5), 42.0);
}

Parameterized Tests

For testing multiple inputs:

class DataTypeTest : public ::testing::TestWithParam<cvc::DataType> {};

TEST_P(DataTypeTest, AllTypes) {
  cvc::DataType type = GetParam();
  cvc::voxels v(cvc::dimension(5, 5, 5), type);
  EXPECT_EQ(v.voxelType(), type);
}

INSTANTIATE_TEST_SUITE_P(
  AllDataTypes,
  DataTypeTest,
  ::testing::Values(
    cvc::UChar, cvc::UShort, cvc::UInt,
    cvc::Float, cvc::Double, cvc::UInt64
  )
);

Multithreaded Testing

TEST(StateTest, ConcurrentValueReads) {
  state::instance(app)("test.concurrent").value("initial");
  
  std::atomic<int> success_count(0);
  std::vector<boost::thread> threads;
  
  for (int i = 0; i < 10; ++i) {
    threads.emplace_back([&success_count]() {
      for (int j = 0; j < 100; ++j) {
        std::string val = state::instance(app)("test.concurrent").value();
        if (!val.empty()) success_count++;
      }
    });
  }
  
  for (auto& t : threads) t.join();
  
  EXPECT_EQ(success_count.load(), 1000);
  state::instance(app)("test").reset();
}

Code Coverage Analysis

Quick Start - Automated Script

The easiest way to generate coverage:

./generate_coverage.sh

This script automatically:

  1. Configures build with coverage enabled
  2. Builds the project
  3. Runs all tests
  4. Generates HTML coverage report
  5. Shows coverage summary

The report will be in build-coverage/coverage_html/index.html.

Prerequisites

lcov comes from cvcpkg, like everything else libcvc builds with: lcov 2.0 on cvcpkg's perl, with the Perl modules it needs. gcov comes with the compiler.

cvcpkg install lcov --prefix ~/cvc-tools --config release --link shared
export PATH="$HOME/cvc-tools/bin:$PATH"   # or pass -DLCOV_PATH/-DGENHTML_PATH

Manual Coverage Generation

# Step 1: Configure with coverage enabled
cmake -B build-coverage -S . \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="--coverage" \
  -DCVC_ENABLE_COVERAGE=ON \
  -DCVC_BUILD_TESTS=ON

# Step 2: Build the project
cmake --build build-coverage -j$(nproc)

# Step 3: Generate coverage report
cmake --build build-coverage --target coverage

# Step 4: View the report
xdg-open build-coverage/coverage_html/index.html

Coverage Targets

  • coverage - Runs tests and generates HTML report
  • coverage-view - Opens the coverage report in a browser

Understanding Coverage Reports

The HTML report shows:

  • Line Coverage: Percentage of code lines executed
  • Function Coverage: Percentage of functions called
  • Branch Coverage: Percentage of conditional branches taken

Color coding:

  • Green: Code executed by tests
  • Red: Code not executed
  • Orange: Partially executed (branches)

Coverage Workflow

  1. Initial Coverage: Run coverage on current tests
  2. Identify Gaps: Look for red (uncovered) sections
  3. Add Tests: Write tests for uncovered code
  4. Re-run Coverage: Verify improvement
  5. Iterate: Repeat until target coverage reached

Coverage Goals

CI enforces a hard line-coverage gate: the package-linux / libcvc-debug job fails when line coverage of the supported in-repo surface drops below COVERAGE_MIN (80%), set at the top of .github/workflows/ci.yml. A PR that adds enough untested code to drag the project below the gate cannot merge without also adding tests.

Additional (advisory) targets:

  • Critical Paths: 100% coverage
  • New code in a PR: cover what you add; don't rely on headroom

Filtering Coverage

The gate and the reports measure the same, repo-only view. The filter is an allowlist of src/ + inc/ (a /usr/* blocklist would still count Boost/CGAL/HDF5 headers from the libcvc-deps prefix that are inlined into instrumented TUs), minus:

  • Test code (*/test/*, */tests/*, *_test.cpp)
  • Vendored XmlRpc++ (src/xmlrpc/)
  • Vendored contour-spectrum code (src/cvc/geometry/cvc-mesher/contour/), per docs/roadmap/COVERAGE_IMPROVEMENT_PLAN.md §4 Option A
  • Vendored stb single-file headers (src/cvc/image/third_party/)

The local coverage CMake target applies the identical filter, so numbers from ./generate_coverage.sh are directly comparable to the CI gate.

Coverage with Specific Tests

cd build-coverage

# Clear previous data
lcov --directory . --zerocounters

# Run specific tests
./bin/state_test --gtest_filter="*Futures*"

# Capture and generate report
# --ignore-errors negative,gcov,mismatch: geninfo can emit a spurious
# negative count for inlined STL symbols (a gcov quirk) and gcov can fail
# on toolchain version skew; neither should abort the capture.
lcov --directory . --capture --output-file coverage.info --ignore-errors negative,gcov,mismatch
lcov --remove coverage.info '/usr/*' '*/test/*' -o coverage_filtered.info
genhtml coverage_filtered.info -o coverage_html

Advanced Testing Features

Multithreaded Tests (12 tests)

The test suite includes comprehensive multithreaded tests to validate concurrent access:

./bin/state_test --gtest_filter="*Concurrent*:*Deadlock*:*Stress*"

Test Coverage:

  1. ConcurrentValueReads - 10 threads, 1,000 reads each
  2. ConcurrentValueWrites - 10 threads writing different nodes
  3. ConcurrentWritesToSameNode - 20 threads, high contention
  4. ConcurrentDataOperations - 8 reader/writer threads
  5. ConcurrentSignalHandling - Signal propagation under load
  6. ConcurrentHierarchyCreation - Parallel tree building
  7. ConcurrentTraversal - Tree traversal during modifications
  8. ConcurrentResetOperations - Reset mixed with read/write
  9. ConcurrentPropertyTreeOperations - Serialization under modifications
  10. DeadlockDetectionValueAndSignal - Reentrancy testing
  11. StateObjectMultithreaded - CRTP pattern with 8 threads
  12. StressTestCombinedOperations - 2,162 operations in 1 second

Futures API Tests (11 tests)

Async programming patterns with the futures API:

./bin/state_test --gtest_filter="*Futures*:*Future*"

Features Tested:

  • Blocking waits for values/data
  • Timeout exceptions (cvc::timeout_error)
  • Callback registration and firing
  • state_future<T> template class
  • Producer-consumer patterns
  • Multiple waiters on same state

Example:

TEST(StateTest, WaitForValue) {
  boost::thread writer([]() {
    boost::this_thread::sleep_for(boost::chrono::milliseconds(100));
    state::instance(app)("test.future.wait").value(12345);
  });
  
  int val = state::instance(app)("test.future.wait").wait_for_value<int>();
  EXPECT_EQ(val, 12345);
  
  writer.join();
}

State Object Pattern Tests (11 tests)

CRTP-based state object pattern for automatic state management:

./bin/state_test --gtest_filter="*StateObject*"

Test Coverage:

  • Configuration management (window size, fullscreen)
  • Data processing (status tracking, error counting)
  • Renderer state (camera, render mode)
  • Application settings (theme, language)
  • External state access via per-app cvc::state::instance(app)
  • Nested state paths
  • Multiple instances with unique state paths
  • Boost::any data storage
  • Async handler execution

CUDA GPU Tests (20 tests)

GPU acceleration testing (requires CUDA-capable GPU):

./bin/voxels_test --gtest_filter="*CUDA*"

Test Categories:

  • Device Management (4 tests)
  • Memory Operations (4 tests)
  • Operations on GPU (4 tests)
  • Algorithms on GPU (2 tests)
  • Integration Tests (3 tests)
  • Large Volume Performance (3 tests)

Stress Testing

BunnyVolumeConvergence - Most intensive test:

  • Progressive resolutions: 32³ → 64³ → 128³ → 256³
  • Volume estimation via interior voxel counting
  • Convergence verification across resolutions
  • ~239 seconds execution time
  • Real-world performance benchmark

Performance Benchmarks

SDF Performance (Stanford Bunny: 34,834 triangles)

Thread-Safe Refactoring (v2.0):

  • Before optimization: ~2600 seconds (43+ minutes) at 256³
  • After optimization: ~234 seconds (3.9 minutes) at 256³
  • Performance gain: 11x speedup 🚀

Optimization Techniques:

  1. Cell reference caching in octree building
  2. Cell reference caching in ray casting
  3. Unconditional BOOST_DISABLE_ASSERTS

Resolution Scaling

Resolution Voxels Time Memory Performance
32³ 32,768 ~0.05s ~10 MB Excellent
64³ 262,144 ~0.2s ~50 MB Excellent
128³ 2,097,152 ~1.5s ~250 MB Good
256³ 16,777,216 ~15s ~1.5 GB Acceptable

Performance scales approximately O(n) for fixed triangle count with optimizations.

Build Type Impact

Build Type Optimization SDF 256³ Time Speedup
Release -O3 -DNDEBUG ~220s 1.0x (baseline)
RelWithDebInfo -O2 -g -DNDEBUG ~234s 0.94x
Debug -g (no opt) ~3027s 0.07x
Debug + Coverage -g --coverage ~3030s 0.07x

Recommendation: Use RelWithDebInfo for development/testing.

CUDA Filter Performance (December 2025)

GDTV Filter CUDA Acceleration:

The GDTV (Gradient-Domain Total Variation) filter shows exceptional GPU speedup due to its fully parallelizable Jacobi iteration pattern:

Configuration CPU Time (ms) GPU Time (ms) Speedup Max Diff Status
32³, 5 iter 229.6 12.8 17.9x 3.05e-05 ✅
64³, 5 iter 917.0 15.1 60.9x 3.05e-05 ✅
32³, 20 iter 352.7 12.5 28.2x 3.05e-05 ✅
64³, 20 iter 3055.6 36.1 84.7x 3.05e-05 ✅
128³, 5 iter 8301.8 138.3 60.0x 3.05e-05 ✅
128³, 10 iter 18741.2 217.8 86.1x 3.05e-05 ✅

Key Insights:

  1. Speedup scales with iterations: More iterations = higher speedup (20 iterations shows 28-85x vs 5 iterations shows 18-61x)
  2. Speedup scales with volume size: Larger volumes = better GPU utilization (128³ > 64³ > 32³)
  3. Perfect numerical accuracy: GPU results match CPU within 3×10⁻⁵ (negligible floating-point differences)
  4. GPU timing includes memory transfer: Speedup would be even higher for pure computation
  5. Best-in-class GPU candidate: GDTV is the most GPU-friendly algorithm in the voxels library

Why GDTV Shows Exceptional Speedup:

  • Both gradient computation AND filtering are GPU-accelerated
  • Each iteration is fully parallelizable (Jacobi pattern - no data dependencies)
  • Iteration count multiplies benefit (20 iterations = 20× parallelizable work)
  • Memory-bound operation benefits from GPU memory bandwidth (350+ GB/s vs ~50 GB/s CPU)
  • Thousands of independent computations per iteration

Bilateral Filter CUDA Acceleration:

Bilateral filter also shows excellent GPU performance:

  • Small volumes (32³): ~17x speedup
  • Medium volumes (64³): ~20x speedup
  • Large volumes (128³): ~25x speedup
  • Numerical accuracy: perfect match with CPU implementation

Contrast Enhancement: NOT CUDA-accelerated

  • Reason: 2/3 of algorithm (propagation phase) is inherently sequential
  • Only 1/3 (stretching phase) is parallelizable - not worth the complexity
  • Documentation explains architectural decision

Implementation History

Initial Implementation (June 2025)

  • 54 tests: Basic coverage of app (25) and state (29) components
  • Google Test v1.14.0 integration via CMake FetchContent
  • Automatic test building (ON by default)
  • CTest integration with custom check target

Expansion Phase (November-December 2025)

Multithreaded Testing (12 tests):

  • Concurrent access validation
  • High contention scenarios (20 threads)
  • Deadlock detection
  • Stress testing (2,162 operations/sec)

Futures API (11 tests):

  • Blocking waits with timeout
  • Callback registration
  • Producer-consumer patterns
  • state_future<T> template class

State Object Pattern (11 tests):

  • CRTP pattern validation
  • Configuration, DataProcessor, Renderer examples
  • External access, multiple instances
  • Async handler execution

Additional Coverage:

  • Volume tests (29)
  • Geometry tests (47)
  • Algorithm tests (6)
  • Voxels tests expanded to 128
  • CUDA GPU tests (20)

Current Status (April 2026)

  • 577 tests: 100% passing
  • 90.5% coverage: On core components
  • Comprehensive documentation: Consolidated TESTING.md
  • Real-world validation: Stanford Bunny, SDF convergence
  • Production-ready: Thread-safe, async patterns, GPU acceleration

Continuous Integration

GitHub Actions Example

name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Install Dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y \
            libboost-all-dev \
            lcov
      
      - name: Configure
        run: |
          cmake -B build \
            -DCMAKE_BUILD_TYPE=Debug \
            -DCVC_BUILD_TESTS=ON \
            -DCVC_ENABLE_COVERAGE=ON
      
      - name: Build
        run: cmake --build build -j$(nproc)
      
      - name: Test
        run: cd build && ctest --output-on-failure
      
      - name: Coverage
        run: cmake --build build --target coverage
      
      - name: Upload Coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./build/coverage.info

Running Tests in CI

# Configure
cmake -B build -DCVC_BUILD_TESTS=ON

# Build
cmake --build build

# Test with output on failure
cd build && ctest --output-on-failure

Troubleshooting

Tests Not Building

Issue: Tests aren't being built

Solution:

# Verify CVC_BUILD_TESTS is ON
cmake -B build -DCVC_BUILD_TESTS=ON

# Check Google Test was fetched
ls build/_deps/googletest-src/

# Clean and rebuild
rm -rf build && cmake -B build && cmake --build build

Tests Failing

Issue: Some tests fail

Solution:

# Run with verbose output
ctest --output-on-failure --verbose

# Run specific test directly
./build/bin/state_test --gtest_filter="*FailingTest*"

# Check for proper cleanup
grep -n "reset()" src/cvc/tests/state_test.cpp

Link Errors

Issue: Undefined references to Boost or dependencies

Solution:

# Verify dependencies are found
cmake -B build -DCMAKE_FIND_DEBUG_MODE=ON

# Check that tests link against cvc library
cat build/src/cvc/tests/CMakeFiles/state_test.dir/link.txt

Coverage Issues

Issue: Coverage data is empty

Solution:

  • Ensure you built with -DCVC_ENABLE_COVERAGE=ON
  • Verify tests ran: cd build && ctest
  • Use Debug build: -DCMAKE_BUILD_TYPE=Debug
  • Check lcov is installed: which lcov

Issue: Low coverage on new code

Solution:

  • Verify tests exercise new code paths
  • Check for untested error handling
  • Add tests for edge cases
  • Use coverage report to identify gaps

Multithreaded Test Issues

Issue: Multithreaded tests hang or timeout

Solution:

  • Check for proper thread cleanup
  • Verify boost::thread::interrupt() support
  • Increase timeout values if needed
  • Run with --gtest_repeat=10 to detect flakiness

CUDA Test Issues

Issue: CUDA tests fail or skip

Solution:

  • Verify CUDA toolkit is installed
  • Check GPU is available: nvidia-smi
  • Ensure CVC_USING_CUDA is defined
  • Tests gracefully skip if CUDA unavailable

Best Practices

1. Test Independence

Each test should be completely independent:

  • No shared global state between tests
  • Proper cleanup after each test
  • Use unique namespaces (test.*)

2. Descriptive Names

Use clear, descriptive test names:

// Good
TEST(StateTest, ConcurrentWritesToSameNode_HighContention)

// Bad
TEST(StateTest, Test5)

3. Arrange-Act-Assert Pattern

Structure tests clearly:

TEST(VoxelsTest, FillOperation) {
  // Arrange
  voxels v(dimension(10, 10, 10), Float);
  
  // Act
  v.fill(42.0);
  
  // Assert
  EXPECT_DOUBLE_EQ(v(5, 5, 5), 42.0);
}

4. Test One Thing

Each test should verify one specific behavior:

// Good - tests one specific operation
TEST(StateTest, ValueSetAndGet)
TEST(StateTest, ValueTypeConversion)

// Bad - tests multiple unrelated things
TEST(StateTest, EverythingAboutValues)

5. Use Appropriate Assertions

Choose the right assertion for the job:

  • EXPECT_* for non-fatal assertions (continues test)
  • ASSERT_* for fatal assertions (stops test)
  • EXPECT_DOUBLE_EQ for floating point
  • EXPECT_THROW for exception testing

6. Keep Tests Fast

Fast tests encourage frequent running:

  • Use smaller datasets when possible
  • Mock expensive operations
  • Run intensive tests separately
  • Use --gtest_filter during development

7. Document Complex Tests

Add comments for non-obvious test logic:

TEST(StateTest, ConcurrentWritesToSameNode) {
  // This test validates thread safety under high contention.
  // 20 threads simultaneously write to the same state node.
  // The final value should be from one of the threads.
  // The test passes if no crashes or deadlocks occur.
  
  // ... test implementation ...
}

Future Improvements

Potential Test Additions

  1. I/O Testing

    • Volume format tests (RAWIV, MRC, Spider, HDF5, VTK)
    • Geometry format tests (OFF, OBJ, RAW variants)
    • Error handling for corrupt files
    • Large file streaming

  2. Algorithm Testing

    • Meshing algorithm validation
    • Isosurfacing edge cases
    • SDF accuracy metrics
    • Performance benchmarks

  3. Integration Tests

    • End-to-end workflows
    • Multi-component pipelines
    • Real application scenarios

  4. Performance Testing

    • Benchmark suite
    • Memory profiling
    • Regression detection
    • Scalability tests

  5. Platform Testing

    • Windows-specific tests
    • macOS validation
    • 32-bit vs 64-bit
    • Different compilers

Infrastructure Improvements

  • Automated nightly tests
  • Performance tracking over time
  • Memory leak detection (Valgrind)
  • Address sanitizer integration
  • Fuzzing for I/O operations
  • Benchmark comparisons
  • Code quality metrics

Conclusion

The libcvc testing infrastructure provides comprehensive validation of core functionality with:

  • ✅ 577 tests with 100% pass rate
  • ✅ 90.5% coverage on core components
  • ✅ Multithreaded validation
  • ✅ Async programming via futures API
  • ✅ GPU acceleration testing (CUDA)
  • ✅ Real-world stress tests (SDF convergence, state_object patterns)
  • ✅ Production-ready thread-safe architecture

The tests are well-organized, thoroughly documented, and easy to extend. The infrastructure supports rapid development with fast feedback cycles and comprehensive coverage analysis.

Key Achievements:

  • Thread-safe SDF v2.0 with 11x performance improvement
  • Comprehensive state management testing (102 tests)
  • State object pattern validation (11 tests)
  • CUDA GPU acceleration coverage (20 tests)
  • Real-world validation (Stanford Bunny, 34,834 triangles)

The testing foundation ensures libcvc is reliable, performant, and ready for production use.