Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,10 @@ canonical multi-architecture version tag.
CMake options (full table in `docs/getting-started.md`):
- `DAQIRI_ENGINE` — space-separated list of optional engines to compile. Valid values: `dpdk` (raw Ethernet) and `ibverbs` (RDMA/RoCE). Linux sockets (UDP/TCP) are always built in, so there is no `socket` value. Default is `"dpdk ibverbs"`.
- `DAQIRI_BUILD_PYTHON` — builds `pybind11` bindings from `python/`.
- `DAQIRI_BUILD_EXAMPLES` — builds the benchmark executables (default `ON`).
- `DAQIRI_BUILD_EXAMPLES` — builds the benchmark executables (default `ON`). The
hardware-free `daqiri_config_validate` tool is always built and installed.
- `BUILD_TESTING` — builds and registers the hardware-free C++ tests under
`tests/cpp/` with CTest (default `ON`). Set it to `OFF` to omit test targets.
- `DAQIRI_BUILD_APPLICATIONS` — builds the end-to-end example applications under `applications/` (default `OFF`; requires TensorRT, e.g. the `BASE_IMAGE=torch` container). Currently builds `applications/resnet50_inference/` (DAQIRI → TensorRT ResNet inference).
- `DAQIRI_ENABLE_OTEL_METRICS` — enables OpenTelemetry metrics instrumentation (default `OFF`).
- `DAQIRI_REORDER_GPU_PROFILE` — enable CUDA event timing in the DPDK reorder kernels (off by default).
Expand All @@ -45,7 +48,20 @@ python3 -m venv .venv
.venv/bin/python -m pytest
```

The default suite collects only `tests/portable/`. Future build-backed C++ tests live under `tests/cpp/`; Python-binding tests live under `tests/bindings/` and require a container built with `DAQIRI_BUILD_PYTHON=ON`. Platform tests live under `tests/platform/` and are selected by CI/CD jobs running on provisioned GPU/NIC systems; they are never part of the default pytest collection. The project container already includes the current test packages; use the container-specific dependency command in `tests/README.md` when `tests/requirements.txt` changes.
Validate checked-in configurations through the production parser and common semantic checks
without initializing hardware:

```bash
python3 scripts/check_daqiri_configs.py --validator build/tools/daqiri_config_validate
```

The default pytest suite collects only `tests/portable/`. Build-backed C++ tests live under
`tests/cpp/` and run through CTest; Python-binding tests live under `tests/bindings/` and
require a container built with `DAQIRI_BUILD_PYTHON=ON`. Platform tests live under
`tests/platform/` and are selected by CI/CD jobs running on provisioned GPU/NIC systems;
they are never part of the default pytest collection. The project container already includes
the current test packages; use the container-specific dependency command in
`tests/README.md` when `tests/requirements.txt` changes.

Integration and performance verification is done via the benchmark executables in `examples/`, driven by YAML configs. Build outputs (`examples/CMakeLists.txt:59-71`):

Expand Down Expand Up @@ -112,7 +128,7 @@ clang-format -style=file -i -fallback-style=none <files>
### Engine abstraction
`src/engine.h` defines `daqiri::Engine` — an (almost) ABC with ~50 virtual methods covering init, RX/TX burst dequeue/enqueue, header-fill helpers, buffer free, socket connection helpers, runtime TCP/UDP `setsockopt` passthrough, and RDMA connection setup. Engines live in `src/engines/<name>/` (`dpdk/`, `rdma/`, `socket/`, `ibverbs/`). `DAQIRI_ENGINE` selects the optional `dpdk` and `ibverbs` engines at CMake configure time; the `socket` engine is always built. The user-facing value `ibverbs` builds **two** internal engines that both use libibverbs: `rdma` (`src/engines/rdma/`, `DAQIRI_ENGINE_RDMA`, RoCE/InfiniBand for socket `roce://`) and `ibverbs` (`src/engines/ibverbs/`, `DAQIRI_ENGINE_IBVERBS`, the pure-DevX MPRQ raw-Ethernet engine). Each engine produces its own static library (`daqiri_dpdk`, `daqiri_rdma`, `daqiri_socket`, `daqiri_ibverbs`) linked into `daqiri_common`, and each adds a `DAQIRI_ENGINE_<NAME>=1` compile definition.

`EngineType` (`include/daqiri/types.h`) is resolved from `(stream_type, engine)`: `raw` defaults to `EngineType::IBVERBS` when that engine is built (falling back to `EngineType::DPDK` in DPDK-only builds); `raw` + `engine: "dpdk"` explicitly selects `EngineType::DPDK`; `socket` + a `roce://` endpoint (or `engine: "ibverbs"`) selects `EngineType::RDMA`. The stream-aware `config_engine_from_string(str, stream_type)` overload encodes the `ibverbs`→`{IBVERBS for raw, RDMA for socket}` split. `EngineFactory` (in `engine.h`) is a singleton that instantiates the active engine. `daqiri_init(...)` resolves which engine to use from the `NetworkConfig` and then delegates everything through the `Engine` vtable. There is only ever **one** active `Engine` per process.
`EngineType` (`include/daqiri/types.h`) is resolved from `(stream_type, engine)`: `raw` defaults to `EngineType::IBVERBS` when that engine is built (falling back to `EngineType::DPDK` in DPDK-only builds); `raw` + `engine: "dpdk"` explicitly selects `EngineType::DPDK`; `socket` + a `roce://` endpoint (or `engine: "ibverbs"`) selects `EngineType::RDMA`. The stream-aware `config_engine_from_string(str, stream_type)` overload encodes the `ibverbs`→`{IBVERBS for raw, RDMA for socket}` split. `EngineFactory` (in `engine.h`) is a singleton that instantiates the active engine. `daqiri_init(...)` resolves which engine to use from the `NetworkConfig`, runs the shared hardware-independent semantic validation, and only then delegates initialization through the `Engine` vtable. The standalone `daqiri_config_validate` tool calls the same parser and shared checks without creating an engine. There is only ever **one** active `Engine` per process.

The always-built socket engine implements Linux UDP/TCP streams directly. Applications that need kernel socket tuning call `socket_setsockopt(conn_id, level, optname, optval, optlen)` after resolving a TCP/UDP connection ID; DAQIRI passes the numeric Linux constants through without maintaining a symbolic option map. `socket_setsockopt` is not supported for `roce://` connections, which delegate to the RDMA/ibverbs path.

Expand Down
6 changes: 6 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ project(daqiri VERSION ${DAQIRI_PROJECT_VERSION} LANGUAGES C CXX CUDA)

include(GNUInstallDirs)
include(CMakePackageConfigHelpers)
include(CTest)
find_package(CUDAToolkit REQUIRED)

set(DAQIRI_ABI_VERSION "2" CACHE STRING "DAQIRI shared library ABI version")
Expand All @@ -55,6 +56,11 @@ option(DAQIRI_ENABLE_OTEL_METRICS "Enable OpenTelemetry metrics instrumentation"
set(DAQIRI_ENGINE "dpdk ibverbs" CACHE STRING "Optional engine implementations to build: dpdk, ibverbs (Linux sockets are always built in)")

add_subdirectory(src)
add_subdirectory(tools)

if(BUILD_TESTING)
add_subdirectory(tests/cpp)
endif()

if(TARGET yaml-cpp)
install(
Expand Down
12 changes: 11 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,22 @@ python3 -m venv .venv

The default invocation collects only `tests/portable/`. Tests importing the compiled
`daqiri` module belong under `tests/bindings/` and require a container built with
`DAQIRI_BUILD_PYTHON=ON`. Future C++ tests belong under `tests/cpp/`. Tests requiring
`DAQIRI_BUILD_PYTHON=ON`. Build-backed C++ tests belong under `tests/cpp/` and run
through CTest. Tests requiring
a GPU, NIC, hugepages, privileged access, or a particular host topology belong under
`tests/platform/` and are selected by dedicated CI/CD jobs on compatible provisioned
runners. See `tests/README.md` for the container dependency command, supported
invocations, and marker policy.

Build `daqiri_config_validate` in the required project container before running
`scripts/check_pr.sh`. The check script validates representative checked-in configurations
through the production C++ parser and hardware-independent semantic checks. Set
`DAQIRI_CONFIG_VALIDATOR` when the executable is not at `build/tools/daqiri_config_validate`.

When a new example config exercises a configuration form these cases do not cover, add a
representative case to `scripts/check_daqiri_configs.py`. Add a focused fixture for any new
validation rule.

#### Pull Requests

Developer workflow for code contributions is as follows:
Expand Down
2 changes: 2 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,8 @@ RUN cmake -S . -B build \
-DDAQIRI_ENABLE_S3=${DAQIRI_ENABLE_S3} \
-DDAQIRI_ENGINE="${DAQIRI_ENGINE}" \
&& cmake --build build -j "$(nproc)" \
&& python3 scripts/check_daqiri_configs.py \
--validator build/tools/daqiri_config_validate \
&& cmake --install build

# ==============================
Expand Down
15 changes: 15 additions & 0 deletions docs/api-reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,21 @@ want to interoperate with existing configuration code.

See `examples/daqiri_bench_*.yaml` for complete working examples.

## Validate without hardware initialization

Use `daqiri_config_validate` to check YAML files before running an application, such as in CI or
on a machine without the target NIC. `daqiri_init()` performs these checks during startup. The
command cannot determine whether the selected engine and hardware support every requested
setting.

```bash
daqiri_config_validate config.yaml another-config.yaml
```

The command exits with status `0` when every file is valid, `1` when any file is invalid, and
`2` when no file was provided. It is built and installed even when
`DAQIRI_BUILD_EXAMPLES=OFF`.

OpenTelemetry metrics do not add YAML fields. Metrics-enabled builds use the
same interface, queue, and flow names from the active configuration as metric
labels, and applications are still responsible for configuring the OpenTelemetry
Expand Down
4 changes: 2 additions & 2 deletions docs/api-reference/cpp.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ daqiri::NetworkConfig config;
auto status = daqiri::daqiri_init(config);
```

After `daqiri_init()` returns `Status::SUCCESS`, all memory regions are allocated, NIC
queues are configured, and worker threads are running.
`daqiri_init()` checks the configuration, prepares packet memory, and starts the selected engine.
It returns `Status::SUCCESS` when DAQIRI is ready.

`MemoryKind::HUGE` is an explicit hugetlb request for DAQIRI-owned memory. DAQIRI does not
substitute regular or transparent-hugepage memory when the requested hugetlb allocation is
Expand Down
3 changes: 2 additions & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,8 @@ DAQIRI's shared-library ABI version is tracked separately through
|--------|---------|-------------|
| `DAQIRI_ENGINE` | `"dpdk ibverbs"` | Space-separated list of optional engine implementations to compile in. Valid values: `dpdk` (Raw Ethernet) and `ibverbs`. `ibverbs` builds two libibverbs-based engines: RDMA/RoCE (for `stream_type: "socket"` with `roce://` endpoints) and the default Mellanox/mlx5 Multi-Packet (striding) Receive Queue engine for `stream_type: "raw"`. Set `engine: "dpdk"` on a raw stream to select the compiled DPDK implementation instead. Linux UDP/TCP sockets are always built in, so there is no `socket` value. |
| `DAQIRI_BUILD_PYTHON` | `OFF` | Build pybind11 Python bindings. |
| `DAQIRI_BUILD_EXAMPLES` | `ON` | Build benchmark executables. |
| `DAQIRI_BUILD_EXAMPLES` | `ON` | Build benchmark executables. The `daqiri_config_validate` tool is always built and installed, including when this option is `OFF`. |
| `BUILD_TESTING` | `ON` | Build and register the hardware-free C++ tests under `tests/cpp/` with CTest. Set this to `OFF` to omit test targets from a production-only build. |
| `DAQIRI_ENABLE_GDS` | `OFF` | Enable cuFile-backed burst file writes from CUDA device memory. Host-memory writes use POSIX APIs without GDS. |
| `DAQIRI_ENABLE_OTEL_METRICS` | `OFF` | Enable OpenTelemetry C++ metrics instrumentation. When enabled, OpenTelemetry C++ API package metadata must be available to CMake. |
| `DAQIRI_ENABLE_S3` | `OFF` | Enable AWS SDK-backed asynchronous raw packet writes to S3. |
Expand Down
3 changes: 3 additions & 0 deletions docs/tutorials/configuration-walkthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ If you don't have any NIC at all, the `*_sw_loopback*` variants of the Raw Ether

(`DAQIRI_ENGINE` at the CMake layer selects which optional engine implementations to compile in. `dpdk` enables the default raw engine, while `ibverbs` enables both the pure-DevX raw engine and `roce://` endpoints. Linux UDP/TCP sockets are always built in. The default build is `dpdk ibverbs`.)

The example configs show the DAQIRI fields and application settings used for different transport
and hardware setups.

For a shorter selection guide, start with the [Benchmarking overview](../benchmarks/index.md). With a stream type in mind, read down the questions below and stop at the first one that matches what you're trying to do. Each section names the YAML, the binary that consumes it, and any platform-specific notes.

??? question "1. I want to measure baseline throughput"
Expand Down
Loading
Loading