#319 - ARP Resolution

docs/api-reference/cpp.md

@@ -883,9 +883,32 @@ workflow sections above show the common call order and ownership rules.
 
 ### Ports, Traffic, Socket, and RDMA
 
+Raw userspace TX must supply a destination MAC because transmitting a completed
+Ethernet frame bypasses the kernel route and ARP processing used by sockets.
+The raw ibverbs engine can perform that lookup explicitly:
+
+```cpp
+char dst_mac[6] = {};
+daqiri::Status status = daqiri::resolve_ipv4_mac(port, "192.0.2.20", dst_mac, 3000);
+if (status != daqiri::Status::SUCCESS) {
+  // Stop before sending data packets.
+}
+```
+
+The result is a snapshot, not a managed DAQIRI neighbor entry. Reuse it for
+packet templates to keep the data path efficient, and call
+`resolve_ipv4_mac()` again if the peer, gateway, route, link, or network
+namespace may have changed. DAQIRI does not refresh previously returned MAC
+addresses in the background. If the port also has DAQIRI RX queues, configure
+`rx.flow_isolation: true` so unmatched ARP traffic stays on the kernel path; a
+TX-only interface with no `rx.queues` needs no `rx` section. See
+[Destination MAC resolution](../concepts.md#destination-mac-resolution) for the
+full ownership model.
+
 | Function | Purpose |
 | --- | --- |
 | `get_mac_addr(port, mac)` | Copy a port MAC address into a six-byte buffer. |
+| `resolve_ipv4_mac(port, dst, mac, timeout_ms)` | Resolve an IPv4 destination's next-hop MAC through the raw ibverbs port's Linux route and ARP table. `dst` may be a host-order `uint32_t` or a dotted-decimal string. |
 | `format_eth_addr(dst, addr)` | Convert a `xx:xx:xx:xx:xx:xx` MAC string into a six-byte buffer. Invalid input zeroes the buffer. |
 | `get_port_id(key)` | Resolve an interface name or PCIe address to a port ID. |
 | `get_num_rx_queues(port_id)` | Return the configured or engine-reported RX queue count. |

docs/api-reference/python.md

@@ -690,9 +690,21 @@ The workflow sections above show the common call order and ownership rules.
 
 ### Ports, Traffic, Socket, and RDMA
 
+Raw userspace TX bypasses the automatic route and ARP processing performed for
+Linux sockets. With the raw ibverbs engine, call
+`resolve_ipv4_mac(port, dst, timeout_ms)` after initialization, then reuse the
+returned MAC in packet templates. This is a one-shot lookup: DAQIRI does not
+cache or refresh the returned value, so the application must re-query after a
+peer, gateway, route, link, or network-namespace change, or according to its
+own failure/refresh policy. Configure `rx.flow_isolation: true` so unmatched
+ARP traffic remains on the kernel path when the port also has DAQIRI RX queues.
+A TX-only interface with no `rx.queues` needs no `rx` section. See
+[Destination MAC resolution](../concepts.md#destination-mac-resolution).
+
 | Function | Purpose |
 | --- | --- |
 | `get_mac_addr(port)` | Return `(Status, "aa:bb:cc:dd:ee:ff")`. |
+| `resolve_ipv4_mac(port, dst, timeout_ms)` | Return `(Status, mac_string)` after resolving a host-order integer or dotted-decimal IPv4 destination through the raw ibverbs port's Linux route and ARP table. The binding releases the GIL while waiting. |
 | `format_eth_addr(addr)` | Return six MAC-address bytes from a `xx:xx:xx:xx:xx:xx` MAC string. Invalid input returns zero bytes. |
 | `get_port_id(key)` | Resolve an interface name or PCIe address to a port ID. |
 | `get_num_rx_queues(port_id)` | Return configured RX queue count for a port. |

docs/benchmarks/raw_benchmarking.md

@@ -5,7 +5,7 @@ hide:
 
 # Raw Ethernet Benchmarking
 
-DAQIRI provides raw Ethernet benchmark applications that use DPDK to drive an NVIDIA NIC directly. This page walks through `daqiri_bench_raw_gpudirect`, the TX/RX loopback config, and the raw Ethernet checks needed before interpreting throughput results.
+DAQIRI provides raw Ethernet benchmark applications that use the DPDK or ibverbs engine to drive an NVIDIA NIC directly. This page walks through `daqiri_bench_raw_gpudirect`, the TX/RX loopback config, and the raw Ethernet checks needed before interpreting throughput results.
 
 Make sure to [build the DAQIRI library](../getting-started.md#build-the-daqiri-library) beforehand.
 
@@ -81,14 +81,14 @@ docker run --rm -it --privileged \
 
 If you have two DGX Sparks cross-cabled p0↔p0 instead of a chassis QSFP loop on one machine, use the `_xhost` configs. Each host runs only its own role, so the YAML on each side configures one port instead of two. Both hosts must already be set up per the [DGX Spark profile](../tutorials/system_configuration.md#dgx-spark-profile), with one adjustment: the `daqiri-tx` (`1.1.1.1/24`) and `daqiri-rx` (`2.2.2.2/24`) nmcli profiles are *split across* the two hosts. Bring up `daqiri-tx` on the TX host's p0 and `daqiri-rx` on the RX host's p0, instead of both on one box.
 
-**Network prerequisite (required for RDMA, recommended for raw).** Assigning `/24` addresses on each host is not enough for the kernel to reach the peer over a direct cable. RDMA-CM uses the kernel stack, so you need a host route and a static neighbor on the cabled port before ping or RoCE will work. Run [`scripts/setup_spark_xhost_net.sh`](https://github.com/nvidia/daqiri/blob/main/scripts/setup_spark_xhost_net.sh) on **both** hosts after bringing up the nmcli profile. See the [cross-host variant](../tutorials/system_configuration.md#cross-host-variant-two-sparks) in System Configuration for the full steps.
+**Network prerequisite.** Assigning `/24` addresses on each host is not enough for the kernel to reach the peer over a direct cable. Install a host route on the cabled port by running [`scripts/setup_spark_xhost_net.sh`](https://github.com/nvidia/daqiri/blob/main/scripts/setup_spark_xhost_net.sh) on **both** hosts after bringing up the nmcli profile. The ibverbs raw engine uses that route and Linux ARP when `eth_dst_addr` is omitted; RDMA-CM uses the same kernel route. See the [cross-host variant](../tutorials/system_configuration.md#cross-host-variant-two-sparks) in System Configuration for the full steps.
 
 ```bash
-# TX host (peer MAC from RX: cat /sys/class/net/enp1s0f0np0/address)
-sudo scripts/setup_spark_xhost_net.sh --role tx --peer-mac <RX_P0_MAC>
+# TX host
+sudo scripts/setup_spark_xhost_net.sh --role tx
 
-# RX host (peer MAC from TX)
-sudo scripts/setup_spark_xhost_net.sh --role rx --peer-mac <TX_P0_MAC>
+# RX host
+sudo scripts/setup_spark_xhost_net.sh --role rx
 
 # Verify on each host before starting benches
 ping -c 3 <peer-ip>    # 2.2.2.2 on TX, 1.1.1.1 on RX
@@ -101,7 +101,7 @@ ip route get <peer-ip> # must name enp1s0f0np0, not lo
 # RX host
 sudo ./daqiri_bench_raw_gpudirect daqiri_bench_raw_rx_spark_xhost.yaml --seconds 30
 
-# TX host (set eth_dst_addr to the RX host p0's MAC first: cat /sys/class/net/enp1s0f0np0/address on the RX host)
+# TX host (the ibverbs config resolves the RX MAC through Linux ARP)
 sudo ./daqiri_bench_raw_gpudirect daqiri_bench_raw_tx_spark_xhost.yaml --seconds 30
 ```
 
@@ -255,15 +255,15 @@ interfaces:
 
 To run the benchmarking application to run a loopback on your system, you'll need to modify the `bench_tx` section which configures the application itself, to create the packet headers, pin the application TX worker, and direct the packets to the NIC. Make sure to remove the template brackets `< >`.
 
-- `eth_dst_addr` with the MAC address (and not the PCIe address) of the NIC interface you want to use for Rx. You can get the MAC address of your `if_name` interface with `#!bash cat /sys/class/net/$if_name/address`:
+- `eth_dst_addr` with the MAC address (and not the PCIe address) of the NIC interface you want to use for Rx. You can get the MAC address of your `if_name` interface with `#!bash cat /sys/class/net/$if_name/address`. Keep this field for DPDK and loopback configurations. A cross-host raw ibverbs configuration may omit it when the TX netdev has a usable IPv4 address and route; the benchmark then resolves `ip_dst_addr` through Linux routing and ARP once before its TX loop:
 - `cpu_core` with the CPU core for the benchmark application's TX thread. This is separate from the DAQIRI TX queue `cpu_core`; use a different isolated core when you have one, or deliberately share when the machine has a tight core budget.
 
 ```yaml hl_lines="3 5"
 bench_tx:
 - interface_name: "tx_port" # Name of the TX port from the daqiri config
   cpu_core: 10              # Benchmark application TX thread affinity
   ...
-  eth_dst_addr: <00:00:00:00:00:00> # Destination MAC address - required when Rx flow_isolation=true
+  eth_dst_addr: <00:00:00:00:00:00> # Explicit MAC for DPDK or loopback
   ...
 ```
 
@@ -274,13 +274,14 @@ bench_tx:
     - interface_name: "tx_port" # Name of the TX port from the daqiri config
       cpu_core: 10              # Benchmark application TX thread affinity
       ...
-      eth_dst_addr: 48:b0:2d:ee:83:ad # Destination MAC address - required when Rx flow_isolation=true
+      eth_dst_addr: 48:b0:2d:ee:83:ad # Explicit MAC for DPDK or loopback
       ...
     ```
 
 ??? info "Show explanation"
 
-    - `eth_dst_addr` - the destination ethernet MAC address - will be embedded in the packet headers by the application. This is required here because the Rx interface above has `flow_isolation: true` (explained in more details below). In that configuration, only the packets listing the adequate destination MAC address will be accepted by the Rx interface.
+    - `eth_dst_addr` - when present, this destination Ethernet MAC is embedded in packet headers without a route or neighbor lookup. When it is absent on raw ibverbs, the benchmark resolves the next-hop MAC from `ip_dst_addr` once before sending. Raw userspace TX does not otherwise invoke Linux routing or ARP because the application supplies complete Ethernet frames. If the interface also has DAQIRI RX queues, use `rx.flow_isolation: true` so ARP remains on the kernel path. A TX-only interface with no `rx.queues` needs no `rx` section.
+    - The benchmark reuses the resolved MAC for the entire run; it does not monitor Linux neighbor changes. Long-running applications should define their own refresh policy and call `resolve_ipv4_mac()` again when a peer, gateway, route, link, or namespace may have changed. See [Destination MAC resolution](../concepts.md#destination-mac-resolution).
     - `cpu_core` - the benchmark application's own TX worker thread affinity. Set the matching `bench_rx.cpu_core` for RX workers too. These app-thread fields are distinct from the DAQIRI queue `cpu_core` values that poll the NIC.
     - We ignore the IP fields (`ip_src_addr`, `ip_dst_addr`) for now, as we are testing on a layer 2 network by just connecting a cable between the two interfaces on our system, therefore having mock values has no impact.
     - You might have noted the lack of a `eth_src_addr` field in this `bench_tx` section. This is because the source Ethernet MAC address can be inferred automatically by the DAQIRI library from the PCIe address of the Tx interface referenced above.

docs/concepts.md

@@ -69,6 +69,43 @@ packets into one pre-posted buffer to avoid per-packet allocation. Set
 Raw Ethernet requires an NVIDIA ConnectX-6 Dx or later NIC.
 Packet pacing and timed transmission require ConnectX-7 or later.
 
+#### Destination MAC resolution
+
+A raw-Ethernet application constructs the complete layer-2 frame in user
+space. Unlike a UDP or TCP socket send, that packet does not pass through the
+Linux IP stack, so Linux does not automatically select a route, identify the
+next hop, run ARP, or fill the destination MAC address. Merely configuring an
+IP destination in the packet header is therefore insufficient: userspace
+packet processing cannot rely on the kernel routing and neighbor tables being
+consulted when the packet is transmitted.
+
+The raw ibverbs engine provides `resolve_ipv4_mac()` as an explicit bridge to
+Linux routing and ARP. After `daqiri_init()`, the application supplies a DAQIRI
+port and IPv4 destination. DAQIRI asks Linux for the route constrained to that
+port's kernel netdev and primary IPv4 source, resolves the gateway when the
+route is off-link or the destination when it is on-link, and returns the
+current six-byte next-hop MAC. An existing confirmed Linux neighbor entry
+returns immediately. DAQIRI refreshes a stale entry before returning it;
+otherwise Linux performs ARP while DAQIRI waits up to the caller's deadline.
+This API is available only for the raw ibverbs engine. DPDK and the socket/RDMA
+engines return `NOT_SUPPORTED`.
+
+Resolution is deliberately a one-shot operation. DAQIRI does not maintain a
+second neighbor cache, start a refresh thread, or change a MAC already placed
+in application packet templates. For efficiency, resolve once and reuse the
+result while the route and peer are stable. The application is responsible for
+calling `resolve_ipv4_mac()` again when its refresh policy requires it—for
+example after a link, peer NIC, gateway, route, or network-namespace change, or
+after a transmission/reachability failure. Re-querying still uses Linux's
+neighbor table and is inexpensive while its entry remains usable.
+
+Automatic ARP also requires the NIC's control traffic to remain visible to the
+kernel. If the raw ibverbs interface has DAQIRI RX queues, configure
+`rx.flow_isolation: true` so DAQIRI installs only explicit application flow
+rules and unmatched ARP traffic continues on the kernel path. A TX-only
+interface with no `rx.queues` installs no DAQIRI RX catch-all and does not need
+an `rx` section.
+
 ### Socket
 
 *YAML:* `stream_type: "socket"`. The specific transport is chosen by endpoint

docs/tutorials/configuration-walkthrough.md

@@ -284,7 +284,7 @@ bench_tx: # (25)!
 22. What to do with packets that match this flow. Existing configs can use the legacy single `action:` map with `type: queue` to send packets to the queue with the given `id`. Use `ids: [0, 1]` instead to enable flow-affine IPv4/UDP five-tuple RSS across a non-empty list of unique configured queues; do not specify both `id` and `ids`. New hardware transform flows use ordered `actions:`; RX transform flows can `vlan_pop` or `tunnel_decap` and must end with `type: queue`. An unchanged five tuple remains on one queue, so roughly even packet counts require many reasonably balanced flows rather than one high-rate flow.
 23. :material-package-variant: List of rules to match packets against. **All** rules must hold for a packet to match the flow. Currently supported keys: `udp_src` / `udp_dst` (UDP source/destination port numbers, integer), `ipv4_len` (full IPv4 packet length in bytes, integer). **Adjust to match your incoming traffic.**
 24. The `bench_rx` section is specific to the benchmark application. It is a list of application worker configs. List entries map to DAQIRI RX queues on the named interface, either by explicit `queue_id` or by queue-list order when `queue_id` is omitted. `cpu_core` pins the benchmark application's RX worker thread. It is separate from the DAQIRI queue `cpu_core` above, and can use the same core only when you intentionally want to share. In this base config there is one RX queue, so there is one entry. Other DAQIRI binaries (e.g. the reorder-quantize bench) may add fields here. See those configs for details.
-25. :material-package-variant: The `bench_tx` section configures the TX side of the benchmark: the benchmark application's TX worker core, packet sizes, and the Ethernet/IP/UDP header fields embedded in outgoing packets. It is a list for the same reason as `bench_rx`: each entry maps to a DAQIRI TX queue on the named interface, either by explicit `queue_id` or by queue-list order when `queue_id` is omitted. The `cpu_core` field pins the application TX thread. The `eth_dst_addr` is required when `flow_isolation` is enabled on the RX interface. The `ip_src_addr` / `ip_dst_addr` are only needed when traffic is routed across subnets. For a direct cable loopback, any value works. The `payload_size`, `header_size`, and UDP ports should match your application's packet format. Hardware TX VLAN push or tunnel encapsulation is configured under `daqiri.cfg.interfaces[].tx.flows`, not in `bench_tx`. Application buffers remain the pre-encap packet.
+25. :material-package-variant: The `bench_tx` section configures the TX side of the benchmark: the benchmark application's TX worker core, packet sizes, and the Ethernet/IP/UDP header fields embedded in outgoing packets. It is a list for the same reason as `bench_rx`: each entry maps to a DAQIRI TX queue on the named interface, either by explicit `queue_id` or by queue-list order when `queue_id` is omitted. The `cpu_core` field pins the application TX thread. An explicit `eth_dst_addr` is used unchanged. With the raw ibverbs engine, it may instead be omitted so DAQIRI resolves `ip_dst_addr` (or its gateway) through the selected port's Linux route and ARP table. When that interface also has DAQIRI RX queues, configure `rx.flow_isolation: true` so ARP remains on the kernel path; a TX-only interface with no `rx.queues` needs no `rx` section. Other engines still require an explicit destination MAC. The `payload_size`, `header_size`, and UDP ports should match your application's packet format. Hardware TX VLAN push or tunnel encapsulation is configured under `daqiri.cfg.interfaces[].tx.flows`, not in `bench_tx`. Application buffers remain the pre-encap packet.
 
 ### Flow steering
 

docs/tutorials/system_configuration.md

@@ -1547,14 +1547,14 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
     - TX host: `sudo nmcli connection up daqiri-tx` only (`1.1.1.1/24` on `enp1s0f0np0`)
     - RX host: `sudo nmcli connection up daqiri-rx` only (`2.2.2.2/24` on `enp1s0f0np0`)
 
-    The `/24` assignment alone does not give the kernel a route to the peer. On **each** host, add a host route and static neighbor for the peer on the cabled port (read the peer's MAC on the other box with `cat /sys/class/net/enp1s0f0np0/address`):
+    The `/24` assignment alone does not give the kernel a route to the peer. On **each** host, add a host route on the cabled port. The ibverbs raw engine can then ask Linux to resolve the peer with ARP when `bench_tx.eth_dst_addr` is omitted:
 
     ```bash
     # TX host
-    sudo scripts/setup_spark_xhost_net.sh --role tx --peer-mac <RX_P0_MAC>
+    sudo scripts/setup_spark_xhost_net.sh --role tx
 
     # RX host
-    sudo scripts/setup_spark_xhost_net.sh --role rx --peer-mac <TX_P0_MAC>
+    sudo scripts/setup_spark_xhost_net.sh --role rx
     ```
 
     Verify before running `_xhost` benchmarks (raw or RDMA):

examples/CMakeLists.txt

@@ -69,7 +69,7 @@ function(link_daqiri_bench target)
   target_compile_features(${target} PRIVATE cxx_std_17)
   if(NOT BUILD_SHARED_LIBS AND UNIX AND NOT APPLE)
     set(DAQIRI_EXAMPLE_STATIC_LIBS daqiri::daqiri)
-    foreach(engine IN ITEMS dpdk socket rdma)
+    foreach(engine IN ITEMS dpdk socket rdma ibverbs)
       if(TARGET daqiri_${engine})
         list(APPEND DAQIRI_EXAMPLE_STATIC_LIBS daqiri_${engine})
       endif()

examples/daqiri_bench_raw_rx_spark_xhost.yaml

@@ -18,6 +18,7 @@ daqiri:
   cfg:
     version: 1
     stream_type: "raw"
+    engine: "ibverbs"
     master_core: 8
     debug: false
     log_level: "info"
@@ -44,7 +45,7 @@ daqiri:
             - "Data_RX_GPU"
         flows:
         - name: "flow_0"
-          id: 0
+          id: 1
           action:
             type: queue
             id: 0