Cnuas Virtual AI HPC Infrastructure

A Complete AI/HPC Datacenter, in Software

Cnuas (pronounced Knoo-us, the Irish word for a cluster) is a fully software-emulated AI and HPC infrastructure. Virtual GPUs, virtual GPU-peer fabric, virtual InfiniBand and RoCE switches and virtual RDMA NICs, all built from scratch in C around QEMU, so unmodified Linux guests run real distributed workloads on simulated silicon.

Documentation REST API Reference White Paper (PDF)
Cnuas reference rack, front view modelled on OCP Open Rack v3
Cnuas reference rack, modelled on OCP Open Rack v3
Drive the whole rack from one tool

One Control Plane, CLI and REST API

cnuas is the unified control plane. A single command surface drives the switch fabric, the GPU fabric, GPUs, RDMA NICs and VM lifecycle, and the same service layer is exposed as a REST API for automation. Every CLI capability is also an API endpoint.

cnuas, unified control plane 
$ cnuas system health
┏━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Component ┃ Available ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hiswitch  │ yes       │
│ hilink    │ yes       │
│ higpu     │ yes       │
│ hinic     │ yes       │
│ vm        │ yes       │
└───────────┴───────────┘

$ cnuas switch set-mode 2 STRICT_IB
{ "status": "ok", "port": 2, "mode": "STRICT_IB" }

$ cnuas vm up vm-a --gpus 1
vm-a running, HiNIC + HiGPU attached
Approach

Real APIs, Emulated Silicon

Cnuas does not use host-based shortcuts like Linux bridges or Soft-RoCE. Instead it builds custom QEMU PCIe devices and their corresponding Linux kernel drivers, so the guest sees a real RDMA HCA and a real GPU on its PCIe bus. From the application point of view it is talking to hardware, even though every byte is moving through software.

  • Custom PCIe Vendor and Device IDs registered for HiNIC and HiGPU
  • MMIO BARs, doorbells and interrupts modelled in QEMU C device code
  • Kernel drivers (hicain_net.ko, hicain_ib.ko) probe the device and register with ib_core
  • Standard libibverbs works unmodified, so ibv_rc_pingpong, perftest and NCCL all run
  • Host UNIX-domain sockets act as the physical cable between QEMU NICs and the virtual switch

Why emulation, not virtualisation

A student cannot bring home a $200,000 InfiniBand switch or a 4x H100 server. Existing Linux virt approaches (bridges, Soft-RoCE) hide the hardware semantics that matter, such as DCB, ECN marking, IB LIDs and GPU peer DMA. Cnuas keeps those semantics in software so what you learn here transfers directly to NVIDIA, AMD or Intel production hardware.

Four Core Components

Each component is its own open-source project under PacketFive. Cnuas ties them together as the integration and orchestration layer.

🔌

HiNIC, Virtual RoCE-IB NIC

PCIe device model exposing standard BARs, doorbells and DMA queues. The companion kernel drivers register a netdev and an RDMA HCA. Guest applications use libibverbs as if it were Mellanox or NVIDIA hardware.

🔀

HiSwitch, Virtual TOR Fabric

Standalone C daemon acting as a top-of-rack switch. An epoll event loop listens on per-port UNIX sockets, classifies frames (Ethernet, RoCEv2 or InfiniBand LRH), and forwards using a per-pipeline forwarding table over a JSON-over-UDS control plane.

🖥️

HiGPU, Virtual GPU

QEMU PCIe device with a SIMT execution model, including Streaming Multiprocessors, tensor units, shared memory and HBM-equivalent device memory. It ships a complete NVIDIA-equivalent userland, with HCC (a CUDA equivalent), hi-smi, an hcc compiler and HiCCL (an NCCL equivalent).

⚡

HiLink, GPU Peer Fabric

Independent GPU-to-GPU interconnect, an NVSwitch equivalent. A VM with a HiGPU but no HiNIC can still do GPU peer DMA. It uses ivshmem as the fast path for cross-VM shared memory.

Programmability, End to End

Configure, set up and interact with the entire system from a scriptable CLI or a REST API. Both front ends call the same service layer, so nothing is CLI-only or API-only.

⌨️

Command-Line Interface

Interactive Rich tables by default, and --json on any command for scripting. Command groups cover every domain of the rack.

  • cnuas switch, RoCE and InfiniBand fabric, 18 commands
  • cnuas fabric, HiLink GPU peer fabric
  • cnuas gpu, HiGPU device control
  • cnuas nic, HiNIC and RDMA device control
  • cnuas vm, virtual-machine lifecycle
  • cnuas system, inventory, health and versions
🌐

REST API Service

A FastAPI service (cnuas api) with OpenAPI docs, so any language or orchestrator can drive the platform over HTTP.

cnuas-api, REST service 
$ cnuas api --host 0.0.0.0 --port 8080
# OpenAPI docs at http://host:8080/docs

$ curl host:8080/api/v1/switch/ports
[{ "port": 0, "mode": "AUTO", "link": "up" },
 { "port": 2, "mode": "STRICT_IB", "link": "up" }]
Swagger UI ↗ ReDoc ↗ Programmability Guide ↗
Topology

Modelled on OCP Open Rack v3

The reference deployment follows the Open Compute Project Open Rack v3, the rack design the modern AI buildout was created around. Two 1OU TOR switches (HiSwitch and HiLink) sit at the top of the rack, above a management host and eight 2OU VM blades acting as GPU compute nodes. The U-heights are visual in the emulator, but the layout maps 1:1 to real ORv3 deployments, making the same diagrams useful for teaching, design and real lab build-outs.

  • 10-port RoCEv2 and InfiniBand virtual TOR (HiSwitch)
  • 8-port GPU peer fabric (HiLink)
  • 8 GPU compute blades, each a QEMU guest with HiNIC and HiGPU
  • DCB (PFC, ETS and ECN) modelled end to end across the fabric

Two independent fabrics

The HiSwitch fabric carries every Ethernet frame, including RoCEv2 and RDMA on UDP 4791, the same path NCCL's IB transport, perftest and ibv_rc_pingpong flow over. The HiLink fabric is independent and carries only GPU-to-GPU peer traffic. A guest can use one, the other, or both, exactly as in a real GPU rack.

Built for Education and Research & Development

Cnuas is the lab environment behind the HiCAIN training programme and a platform for open infrastructure research. Students and researchers run real distributed-training workloads, RDMA microbenchmarks and MPI collectives against the emulated fabric, building the skills modern AI cluster operations demand, on a laptop.

🎓

Accessible Lab

No physical GPUs, NICs or switches required. A multi-node GPU cluster fits on a single workstation.

🔬

Realistic Semantics

IB LIDs, GRH and LRH headers, DCB pause frames and ECN marking, all the things that matter for real production debugging.

🛠️

Research Platform

Open-source, modifiable and reproducible. Ideal for academic networking and HPC research where commercial silicon is a black box.

Cnuas on GitHub ↗ Documentation HiCAIN Training ↗