Cnuas Virtual AI HPC Infrastructure Datasheet¶
Part: Cnuas Virtual AI HPC Infrastructure · Type: Software-emulated virtual AI and HPC infrastructure · Version: Platform 2026.07 (`fb4638b-dirty`) · Status: Active development · Product: packetfive.com/cnuas
1. Overview¶
Cnuas (Irish Gaeilge for "cluster") is the Cnuas Virtual AI HPC Infrastructure, a 100% software-emulated virtual AI and HPC infrastructure for teaching and developing HPC, AI, and RDMA networking (RoCEv2 and native InfiniBand) without physical GPU clusters or InfiniBand switches. Every device is a QEMU/userspace model presented to unmodified guest software, so a virtual GPU or NIC behaves like the real part it emulates. Cnuas is developed under the internal HiCAIN engineering programme.
It composes four emulated device families, HiSwitch (TOR fabric switch), HiNIC (RDMA NIC), HiGPU (SIMT accelerator), and HiLink (GPU fabric), over a forked QEMU and a forked Linux kernel, orchestrated as VMs on a single Ubuntu management host.
Key features¶
- Full RDMA fabric, RoCEv2 and native InfiniBand, end to end.
- SIMT virtual GPUs with a CUDA-style runtime and multi-architecture SIMD backends.
- Separate GPU-to-GPU fabric (HiLink) modelling an NVSwitch/NVLink-class interconnect.
- In-switch Subnet Manager, the fabric comes up without external
opensm. - Runs on a single commodity x86-64 host; no special hardware required.
- Consistent management plane spanning CLIs, JSON control sockets, and a browser dashboard.
- Multi-rack topology modelling on the Open Compute Project Open Rack v3 reference.
2. Rack-level block diagram¶
Two-rack reference deployment. Each rack has a HiSwitch top-of-rack (TOR) for the RoCE/IB fabric and a rack-local HiLink switch for the GPU fabric. VMs attach to both fabrics; racks are joined by an inter-TOR link.
flowchart LR
subgraph RACK_A["Rack A (ORv3, 24 OU)"]
direction TB
TOR_A["hicain-vswitchd A<br/>(HiSwitch TOR)"]
LINK_A["higpu-link-switchd A<br/>(HiLink, rack-local)"]
VMA0["vm-a0"] --- TOR_A
VMA1["vm-a1"] --- TOR_A
VMA2["vm-a2"] --- TOR_A
VMA0 -. GPU peer .- LINK_A
VMA1 -. GPU peer .- LINK_A
VMA2 -. GPU peer .- LINK_A
end
subgraph RACK_B["Rack B (ORv3, 24 OU)"]
direction TB
TOR_B["hicain-vswitchd B<br/>(HiSwitch TOR)"]
LINK_B["higpu-link-switchd B<br/>(HiLink, rack-local)"]
VMB0["vm-b0"] --- TOR_B
VMB1["vm-b1"] --- TOR_B
VMB2["vm-b2"] --- TOR_B
VMB0 -. GPU peer .- LINK_B
VMB1 -. GPU peer .- LINK_B
VMB2 -. GPU peer .- LINK_B
end
TOR_A ===|inter-TOR fabric link| TOR_B
classDef tor fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef link fill:#fce7f3,stroke:#9d174d,color:#831843
classDef vm fill:#fef3c7,stroke:#b45309,color:#78350f
class TOR_A,TOR_B tor
class LINK_A,LINK_B link
class VMA0,VMA1,VMA2,VMB0,VMB1,VMB2 vm
2.1 Rack elevation (ORv3, 24 OU)¶
Logical OU (OpenU) slot assignment for a single rack. HiSwitch (TOR) and the rack-local HiLink switch occupy the top OUs; compute VMs fill the remainder. Two such racks form the reference deployment above.
flowchart TB
subgraph RACK["Rack A — Open Rack v3 (24 OU)"]
direction TB
OU24["OU 24 — HiSwitch TOR (hicain-vswitchd) — RoCE/IB fabric"]
OU23["OU 23 — HiLink switch (higpu-link-switchd) — GPU fabric"]
OU22["OU 22 — Management / control plane (hicain-webui, hicain-cli)"]
OU21["OU 20-21 — Power shelf / busbar (ORv3)"]
C1["OU 16-19 — compute vm-a0 (HiNIC + HiGPU)"]
C2["OU 12-15 — compute vm-a1 (HiNIC + HiGPU)"]
C3["OU 08-11 — compute vm-a2 (HiNIC + HiGPU)"]
C4["OU 04-07 — compute vm-a3 (HiNIC + HiGPU)"]
C5["OU 01-03 — storage / spare"]
OU24 --> OU23 --> OU22 --> OU21 --> C1 --> C2 --> C3 --> C4 --> C5
end
classDef tor fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef link fill:#fce7f3,stroke:#9d174d,color:#831843
classDef mgmt fill:#ede9fe,stroke:#6d28d9,color:#4c1d95
classDef pwr fill:#e5e7eb,stroke:#4b5563,color:#1f2937
classDef comp fill:#fef3c7,stroke:#b45309,color:#78350f
class OU24 tor
class OU23 link
class OU22 mgmt
class OU21,C5 pwr
class C1,C2,C3,C4 comp
3. Host / node block diagram¶
Inside a single physical host, the switch daemons run in host userspace; each VM
is a QEMU process exposing a hicain-vnic (NIC) and a higpu (GPU) PCIe device.
Devices connect to the daemons over UNIX-domain sockets (UDS).
flowchart TB
subgraph HOST["HOST — hicain1 (bare-metal, Ubuntu 24.04)"]
direction TB
subgraph DAEMONS["Userspace daemons"]
direction LR
HISWITCH["hicain-vswitchd<br/>(HiSwitch)<br/>10 SEQPACKET ports<br/>FDB + classifier + DCB + SM<br/>/var/run/hicain/"]
HILINK["higpu-link-switchd<br/>(HiLink)<br/>8 SEQPACKET ports<br/>NVSwitch-equivalent<br/>/var/run/hilink/"]
end
subgraph VMA["QEMU vm-a"]
direction TB
VMA_NIC["-device hicain-vnic<br/>(RoCE-IB NIC)"]
VMA_GPU["-device higpu<br/>(GPU + HiLink endpoint)"]
VMA_KMOD["Guest kernel:<br/>hicain_net.ko + hicain_ib.ko<br/>higpu.ko"]
VMA_USR["Guest userspace:<br/>libhicain-rdmav34.so<br/>libhicart.so<br/>perftest / pingpong / apps"]
VMA_NIC --- VMA_KMOD
VMA_GPU --- VMA_KMOD
VMA_KMOD --- VMA_USR
end
subgraph VMB["QEMU vm-b"]
direction TB
VMB_NIC["-device hicain-vnic"]
VMB_GPU["-device higpu"]
VMB_KMOD["Guest kernel:<br/>hicain_net.ko + hicain_ib.ko<br/>higpu.ko"]
VMB_USR["Guest userspace:<br/>libhicain-rdmav34.so<br/>libhicart.so"]
VMB_NIC --- VMB_KMOD
VMB_GPU --- VMB_KMOD
VMB_KMOD --- VMB_USR
end
HISWITCH ===|UDS port_0| VMA_NIC
HISWITCH ===|UDS port_1| VMB_NIC
HILINK -.->|UDS port_0| VMA_GPU
HILINK -.->|UDS port_1| VMB_GPU
end
classDef daemon fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef vm fill:#fef3c7,stroke:#b45309,color:#78350f
classDef nic fill:#dcfce7,stroke:#166534,color:#14532d
classDef gpu fill:#fce7f3,stroke:#9d174d,color:#831843
class HISWITCH,HILINK daemon
class VMA,VMB vm
class VMA_NIC,VMB_NIC nic
class VMA_GPU,VMB_GPU gpu
4. Guest software stack¶
Per-VM stack, from application down to the QEMU device backends. The RDMA path
uses stock libibverbs, which dlopen()s the HiNIC provider; the GPU path uses
libhicart over the higpu.ko char device.
flowchart TB
subgraph VM["VM — vm-a (Ubuntu 24.04 guest)"]
direction TB
subgraph APP["Application layer"]
direction LR
APP_PERF["perftest, pingpong"]
APP_USER["user code:<br/>MPI / libfabric / NCCL / UCX"]
end
subgraph LIB["Userspace libraries"]
direction LR
LIB_VERBS["libibverbs.so.1<br/>(stock rdma-core)"]
LIB_PROV["libhicain-rdmav34.so<br/>(HiNIC provider)"]
LIB_CART["libhicart.so + libhi.so<br/>(HiGPU runtime)"]
end
subgraph KMOD["Guest kernel modules"]
direction LR
KMOD_NET["hicain_net.ko<br/>netdev driver"]
KMOD_IB["hicain_ib.ko<br/>IB-core RDMA driver"]
KMOD_GPU["higpu.ko<br/>GPU char driver<br/>/dev/higpu0"]
end
subgraph QEMU["QEMU device backends"]
direction LR
QEMU_NIC["hicain_vnic.c<br/>PCIe + MMIO + IRQ<br/>UDS to HiSwitch"]
QEMU_GPU["higpu PCI device<br/>+ HiLink endpoint<br/>UDS to HiLink"]
end
APP_PERF --> LIB_VERBS
APP_USER --> LIB_VERBS
APP_USER --> LIB_CART
LIB_VERBS --> LIB_PROV
LIB_PROV --> KMOD_IB
LIB_VERBS --> KMOD_IB
LIB_CART --> KMOD_GPU
KMOD_IB --> KMOD_NET
KMOD_NET --> QEMU_NIC
KMOD_GPU --> QEMU_GPU
end
classDef app fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
classDef lib fill:#e0e7ff,stroke:#3730a3,color:#1e1b4b
classDef kmod fill:#fef3c7,stroke:#92400e,color:#78350f
classDef qemu fill:#dcfce7,stroke:#166534,color:#14532d
class APP_PERF,APP_USER app
class LIB_VERBS,LIB_PROV,LIB_CART lib
class KMOD_NET,KMOD_IB,KMOD_GPU kmod
class QEMU_NIC,QEMU_GPU qemu
5. Platform specifications¶
| Parameter | Value |
|---|---|
| Emulation model | QEMU custom PCIe devices + userspace switch daemons |
| Host OS | Ubuntu 24.04 LTS (x86-64) |
| Guest kernel | Linux fork hicain-v6.19.0 (6.19.0-hicain+) |
| Hypervisor | QEMU fork hicain-qemu-v0.1.0 |
| Inter-VM data transport | UNIX domain sockets (SOCK_SEQPACKET); ivshmem shared memory (GPU fast path) |
| Management transport | JSON over UNIX/TCP sockets |
| Reference topology | Open Compute Project Open Rack v3 (ORv3), up to 24 OU |
| Fabric switch ports | 10 per HiSwitch (8 fabric + uplink + console) |
| GPU fabric ports | 8 per HiLink (configurable, max 32) |
| Build system | Bazel (daemons/CLI/WebUI/packaging), Make (kernel, QEMU) |
| Languages | C (GNU C, daemons + kernel), Python 3.11+ (tooling), C++ (QEMU models) |
| Versioning | Semantic versioning per component (hicain-vMAJOR.MINOR.PATCH) |
6. Component summary¶
| Component | Part / daemon | Kernel | Userspace | Datasheet |
|---|---|---|---|---|
| Fabric switch | hicain-vswitchd |
, (host daemon) | hicain-cli, WebUI |
HiSwitch |
| RDMA NIC | hicain-vnic |
hicain_net.ko, hicain_ib.ko |
libhicain-rdmav34.so |
HiNIC |
| GPU | higpu |
higpu.ko (/dev/higpuN) |
libhicart.so, libhi.so, hicasmi |
HiGPU |
| GPU fabric | higpu-link-switchd |
, (host daemon) | hilink-cli |
HiLink |
6.1 Component versions¶
Live from each submodule's git tag at documentation build time:
| Component | Version (git tag) |
|---|---|
| VDC (superproject) | fb4638b-dirty |
| HiSwitch | v0.1.0-4-g1700b86 |
| HiNIC | v0.1.0-23-gd97b1c9 |
| HiGPU | v0.2.0-3-g4104982 |
| HiLink | v0.1.0 |
| Linux kernel fork | hicain-v6.19.0 |
| QEMU fork | hicain-qemu-v0.1.0-2-g6b326b46bc |
7. Management and control plane¶
| Interface | Description |
|---|---|
hicain-cli |
Switch/fabric management CLI (port status, SM, DCB) |
hilink-cli |
HiLink GPU fabric management CLI |
hicasmi / hi-smi |
GPU inventory and telemetry (nvidia-smi style) |
hicain-tools |
VM lifecycle: image/package build, vm up/down/ssh/console/lab |
hicain-webui |
Browser dashboard (FastAPI backend + web frontend) |
| Control sockets | JSON over UNIX socket (/var/run/hicain/, /var/run/hilink/) |
8. Host requirements¶
| Requirement | Minimum | Recommended |
|---|---|---|
| CPU | x86-64 with AVX2 | AVX-512 (for HiGPU compute) |
| Memory | 16 GB | 64 GB+ (scales with VM count) |
| Virtualization | KVM enabled | KVM + nested virt |
| OS | Ubuntu 24.04 LTS | Ubuntu 24.04 LTS |
| Toolchain | GCC (GNU C), Make, Bazel, Python 3.11+, pytest |
+ mkdocs-material for docs |
9. Roadmap summary¶
Feature availability at a glance. Full Epic/Task breakdown, status, and the GitHub Projects board are in the Product Roadmap.
| Workstream | Available today | Planned |
|---|---|---|
| HiNIC (RDMA NIC) | RoCEv2 + native IB, RC/UD/SMI/GSI, atomics, SRQ, multicast, ICRC, in-switch SM interop | SA path-record, PMA/perfquery, opensm compat |
| HiSwitch (fabric) | 10-port RoCE/IB, DCB, FDB/LFT, in-switch Subnet Manager, telemetry | Switch-side SA/PMA alignment |
| HiGPU (accelerator) | SIMT runtime, libhicart/libhi, AVX2/AVX-512 SGEMM, multi-GPU | Backend dlopen split, device-backend refactor, HiIR, FPGA |
| HiLink (GPU fabric) | Frame protocol, FDB, discovery, sysfs link state, CLI | Collective offload, bandwidth modelling |
| Platform / tooling | VM lifecycle CLI, packaging, submodule split, CI | Documentation pass |
10. Ordering / integration information¶
| Item | Value |
|---|---|
| Superproject repo | PacketFive/vdc (git submodule superproject) |
| Submodules | linux, qemu, src/hinic, src/hiswitch, src/higpu, src/hilink |
| License | Apache-2.0 with custom amendment rider (see repository LICENSE) |
| Quick start | ./build_and_test.sh |
| Documentation | https://packetfive.github.io/vdc/ |
11. Revision history¶
| Revision | Date | Notes |
|---|---|---|
| A | 2026-07-05 | Initial platform datasheet |
| B | 2026-07-05 | Added rack/host/stack diagrams, detailed specs, component driver map |
| C | 2026-07-05 | Added rack elevation, live version table, roadmap summary |