Projects · Security · Simulators · Stack · Proof of work · Runnable today
I’m Naresh, a distributed systems and GPU networking engineer. I work on the infrastructure that large-scale training and inference run on: kernel networking, RDMA/RoCE, DPDK, SR-IOV, KVM and NCCL. When a 512-GPU job stalls, the cause is usually one bad link, NIC or GPU, while every rank reports the same timeout. I build the infrastructure that finds that one component and gets training back fast, and that moves bytes between GPUs without wasting the hardware.
I also design it to be secure by construction. The fastest paths in an AI cluster are the least watched: RDMA bypasses the host kernel, and AI agents now hold the credentials that reach it.
What I'd own on your team
- Make failures attributable. Turn "all 512 ranks timed out" into "rank 3,
mlx5_2, bad optic", then take the cheapest recovery that is still safe. - Keep the fabric fast under pressure. RoCE congestion control (PFC, ECN, DCQCN), rail-optimized topologies, KV-cache transport for disaggregated inference.
- Close the blind spots. Kernel-bypass traffic, agent credentials, and the checkpoint and model supply chain.
Everything here is built in public. A number only appears once it’s measured and linked to its run.
|
goodput: fault attribution & recovery for multi-node training Problem. One bad GPU, NIC, cable or switch port stalls a collective, and every rank times out the same way. Stock tooling detects in minutes, attributes nothing, and restarts the whole job. Approach. Fault injection with a ground-truth ledger, NCCL flight-recorder signatures, DCGM and NIC telemetry joined on the topology to name the culprit rank and component, then the smallest safe recovery: in-process communicator re-init, single-node replacement, peer in-memory checkpoints. Status. Downstream. goodput's labeled incidents train one Hugging Face model, a Qwen3 + LoRA root-cause tier that cites its evidence, demoed in a Space with a signature-matching fast tier (see below). |
kvwire: RDMA KV-cache transport for disaggregated inference Problem. Prefill is compute-bound and decode is memory-bound. Colocated, they interfere: prefills spike TPOT, decodes spike TTFT. Approach. Split them onto separate GPU pools, and make the part in between fast and safe: a GPUDirect RDMA transport, a Triton KV re-layout kernel (kept only if the data justifies it), and a coordinator that stays correct when the network fails mid-transfer. Plugs into vLLM through its KV-connector interface. Status. Downstream. kvwire's measured runs become a public Hugging Face dataset of RDMA KV-transfer traces, explored in a Space that answers "move the KV cache or recompute it?" from measured rows only (see below). |
In NCCL, nearly every hardware fault ends the same way: a watchdog timeout on every rank. The engineering is in the last two columns: naming the one component, then choosing the cheapest recovery that is still safe.
| Fault | What every rank reports | Evidence that names the component | Smallest safe recovery |
|---|---|---|---|
| GPU off the bus (Xid 79) | Watchdog timeout, same collective sequence number | Xid in kernel log / DCGM; the one rank missing from the flight recorder's last completed op | Swap in a spare node, restore from peer in-memory checkpoint |
| Uncorrectable ECC / row-remap failure (Xid 48, 64) | Hang or NaN loss, then timeout | DCGM ECC and row-remap counters on one GPU | Drain node, resume from last checkpoint |
| NVLink fault (Xid 74) | Intra-node collectives stall | Xid 74, NVLink CRC / replay counters | Swap node |
| RoCE link flap or bad optic | Inter-node all-reduce stalls on all ranks | link_downed, symbol / CRC errors on one NIC port, IB async events |
Communicator re-init once the link recovers; else swap node |
| PFC pause storm | All ranks slow, no errors anywhere | Pause counters traced hop by hop to the port that paused first (FIG. 3 below) | Fix the origin (ECN / DCQCN thresholds, isolate port); restarting changes nothing |
| Straggler (thermal / power throttling) | Step-time skew, no timeout | DCGM clock-throttle reasons, per-rank step time | Cordon at the next checkpoint boundary |
An AI agent with a kubeconfig is a remote shell that takes instructions from whatever it reads. One injected sentence in a log line, ticket or web page is enough. No exploit is needed, because the agent's permissions are the blast radius. Underneath it, the GPU fabric is where host security sees least: RDMA and GPUDirect bypass the kernel, so host firewalls, eBPF and host IDS never see that traffic, and NCCL has no built-in encryption or peer authentication. In a cluster where model weights are the crown jewels, I treat security as a property of the infrastructure, not a layer added afterwards.
| Layer | What goes wrong | Why AI makes it worse | Control I design for |
|---|---|---|---|
| Agents & tools | Prompt injection steers an agent that can run kubectl, sbatch or cloud APIs |
The agent reads untrusted text all day and acts at machine speed | Read-only tools by default. Mutations go through an allowlisted, audited executor; destructive actions need a human |
| Identity | Long-lived tokens, shared service accounts, ambient admin | One leaked token reaches every node, bucket and checkpoint | Workload identity, short-lived per-task credentials, deny by default |
| Fabric (RDMA / RoCE / NCCL) | Kernel bypass hides traffic from host controls. Collectives cross the fabric in plaintext. A leaked rkey grants remote memory access |
Gradients, activations and KV cache are the model, in transit | Per-tenant isolation (IB partition keys, VRF per tenant), scoped and revocable memory windows, inline MACsec/IPsec on NICs that offload it, telemetry at the NIC/DPU because the host can't see it |
| Multi-tenant inference | Shared prefix caches leak prompts through timing. GPU memory outlives the job | Serving is multi-tenant by default | Per-tenant cache isolation, scrub on reassignment, MIG or confidential computing (Hopper/Blackwell CC mode) where the threat model needs it |
| Artifacts | Pickled checkpoints run code on load. Unsigned weights and datasets | One tampered checkpoint is restored onto every replica | safetensors / weights_only=True, signed and hashed checkpoints, dataset provenance |
| My own tooling | goodput's RCA model is itself an agent touching production | Same risk as above, on the infrastructure itself | It recommends, never acts: it sees read-only evidence, and recovery runs only through goodput's allowlisted executor planned |
Each GitHub project feeds one Hugging Face project, and nothing is published until the repo has produced real, labeled data.
| Artifact | Built from | What it does | Stage |
|---|---|---|---|
nccl-incident-rca · Space |
goodput | Fast tier: root-cause an NCCL/RDMA incident by signature matching over flight-recorder, Xid and NIC evidence | status page · awaiting goodput M3 |
qwen3-nccl-rca-lora · model |
goodput | Slow tier: structured verdict (fault class, culprit, cited evidence ids), scored automatically against the fault ledger. Ships only if it beats goodput's rules, the zero-shot base model and a majority baseline. Its labeled incidents ship inside the model repo | model card (no weights yet) |
rdma-kv-transfer-traces · dataset |
kvwire | Measured KV-cache transfers over RDMA, each next to its perftest ceiling, plus prefill timings and per-layer timelines. CC-BY-4.0 | dataset card · v0 after kvwire's ceiling sweep |
disagg-kv-explorer · Space |
kvwire | Efficiency vs message size, transfer vs recompute, break-even and exposed transfer, from measured rows only | status page · awaiting data |
Each one runs in the browser, with its model and assumptions stated. No hand-waved benchmarks.
| Model | What it shows | |
|---|---|---|
| 💥 | One dead link, 512 idle GPUs | Cut a link in a live ring all-reduce and watch every rank time out the same way |
| 🔎 | Triaging an NCCL timeout | Four incident drills: find the culprit from flight-recorder, Xid and NIC evidence |
| 🧊 | PFC: the lossless network that can freeze itself | Push a toy RoCE fabric into deadlock, then turn on ECN |
| 🛤️ | Where NCCL traffic actually goes | Rail-optimized vs fat-tree for DP, TP and MoE all-to-all, with and without PXN |
| 📦 | The KV-cache bill | Size the prefill→decode handoff for real model configs and link speeds |
| 💾 | How often should a 16K-GPU job checkpoint? | Young/Daly with sliders: which lever buys the most goodput |
| 🧵 | The packet path | Step one packet through the kernel stack, XDP, DPDK and RDMA |
No self-ratings: every row links to the work and states how far along it is.
| Area | Evidence | Stage |
|---|---|---|
| Collectives & NCCL | goodput · distributed-training-framework-nccl · ring all-reduce post | design + harness |
| RDMA / RoCE / PFC | nicprof · kvwire · PFC post | runnable |
| Linux kernel performance | kptk | runnable |
| Inference serving | kvwire · high-performance-llm-inference-engine · KV-cache post | early |
| GPU kernels (CUDA / Triton) | kvwire KV re-layout kernel · custom-cuda-fused-attention-triton | early |
| eBPF / XDP / DPDK | kernel-level-ai-traffic-shaper · packet path post | early / write-up |
| Secure AI infrastructure | threat model and controls · goodput's read-only RCA tier and allowlisted recovery executor | design |
| Incident triage models | qwen3-nccl-rca-lora (grounded NCCL root-cause analysis, verifiable rewards) · nccl-incident-rca Space | model card (no weights yet) |
| KV-transfer measurements | rdma-kv-transfer-traces · disagg-kv-explorer Space | planned (after kvwire's ceiling sweep) |
- nicprof: which NIC counter explains which millisecond of training step time. Aligns RDMA/ethtool/SR-IOV counters to each rank’s steps and traces PFC pause back to its origin, so a victim port isn’t blamed.
nicprof demo· 69 tests - kptk: explains why a Linux workload is slow from kernel evidence: run-queue delay, NUMA locality, THP fallback, and
perf_event_open(2)called directly. Standard library only · 43 tests
Building training or inference clusters that have to be fast and safe? Let’s talk.
I’m looking for GPU cluster networking, distributed-training infrastructure, inference-platform and AI-infrastructure security roles at AI labs and infra teams.
| Role | Proof of work |
|---|---|
| GPU cluster networking | nicprof: NIC counters to step time ↗ |
| Distributed-training infrastructure | goodput: fault attribution & recovery ↗ |
| Inference platforms | kvwire: RDMA KV-cache transport ↗ |
| AI-infrastructure security | Field guide: AI-infra breaches → |
Open to roles in: USA · Canada · Europe