HiCAIN RoCE-IB-vNIC Driver & Userspace Design¶

1. Overview¶

This document describes the Linux kernel drivers and userspace components needed to make the HiCAIN RoCE-IB-vNIC fully functional for both RoCEv2 and InfiniBand traffic. The goal is that standard, unmodified tools (ethtool, ibstat, ibv_rc_pingpong, perftest) work against our virtual NIC, the student never knows the hardware is emulated.

1.1 Component Stack¶

flowchart TB subgraph US["Userspace Applications"] direction TB APPS["ibv_rc_pingpong  •  ib_send_bw  •  Custom RDMA apps"] VERBS["libibverbs (rdma-core)"] PROV["libhicain.so — HiCAIN userspace provider"] APPS --> VERBS --> PROV end subgraph KERN["Kernel"] direction TB IBCORE["ib_core subsystem"] IBKO["hicain_ib.ko — RDMA provider driver<br/>Implements: ib_device_ops"] NETKO["hicain_net.ko — Network device driver<br/>Implements: net_device_ops, ethtool_ops<br/>PCI probe: vendor=0x1ED5, device=0xCA10"] IBCORE --> IBKO --> NETKO end subgraph QEMU["QEMU"] VNIC["hicain_vnic.c — PCIe device model<br/>UNIX SOCK_SEQPACKET → Virtual Switch"] end PROV -->|"/dev/infiniband/uverbsN"| IBCORE NETKO -->|"MMIO BAR0"| VNIC classDef user fill:#dbeafe,stroke:#1e40af,stroke-width:2px,color:#1e3a8a; classDef kernel fill:#fce7f3,stroke:#9d174d,stroke-width:2px,color:#831843; classDef qemu fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f; class APPS,VERBS,PROV user; class IBCORE,IBKO,NETKO kernel; class VNIC qemu;

1.2 Two Kernel Modules, One PCI Device¶

Module	Role	Registers With
`hicain_net.ko`	PCI probe, MMIO mapping, `net_device`, `ethtool_ops`	PCI subsystem, network stack
`hicain_ib.ko`	RDMA verbs, QP/CQ/MR management, IB transport	`ib_core` subsystem

hicain_net.ko owns the PCI device and provides the transport layer (TX/RX of raw frames). hicain_ib.ko builds on top of it to provide RDMA semantics. This mirrors the real-world split (e.g., mlx5_core + mlx5_ib).

2. Kernel Network Driver, `hicain_net.ko`¶

2.1 PCI Probe¶

The driver matches our QEMU device by vendor/device ID:

static const struct pci_device_id hicain_pci_ids[] = {
    { PCI_DEVICE(0x1ED5, 0xCA10) },
    { 0 },
};

On probe: 1. Enable PCI device, request MMIO region 2. Map BAR0 (4KB register space) 3. Set up MSI interrupt 4. Allocate and register struct net_device 5. Read MAC address from REG_MAC_LO/HI 6. Set link state from REG_LINK_STATUS

2.2 net_device_ops¶

Operation	Implementation
`ndo_open`	Enable interrupts (set IRQ_MASK), set link up
`ndo_stop`	Disable interrupts, set link down
`ndo_start_xmit`	Write skb data to DMA buffer, set TX_ADDR/TX_LEN, ring TX_DOORBELL
`ndo_get_stats64`	Return packet/byte counters
`ndo_set_mac_address`	Update MAC registers

2.3 TX Path¶

1. Kernel calls ndo_start_xmit(skb)
2. Driver copies skb->data to a DMA-coherent TX buffer
3. Write buffer physical address → REG_TX_ADDR_LO/HI
4. Write skb->len → REG_TX_LEN
5. Write any value → REG_TX_DOORBELL (triggers QEMU send)
6. QEMU reads buffer via DMA, sends over UNIX socket to switch
7. IRQ fires (TX_COMPLETE), driver frees skb

2.4 RX Path¶

1. QEMU receives frame from switch via UNIX socket
2. QEMU writes frame to guest DMA buffer, sets RX_LEN, raises IRQ
3. Driver IRQ handler reads REG_RX_STATUS (== RX_STATUS_READY)
4. Driver allocates skb, copies data from DMA RX buffer
5. Driver calls netif_rx(skb) to deliver to network stack
6. Driver writes 0 → REG_RX_STATUS to acknowledge

2.5 ethtool_ops¶

Standard ethtool integration, no ethtool modifications needed:

ethtool Command	Callback	What it Returns
`ethtool -i ethX`	`get_drvinfo`	driver="hicain_net", version, bus_info
`ethtool ethX`	`get_link_ksettings`	Speed 100000 (100Gbps emulated), Full duplex
`ethtool ethX`	`get_link`	Link status from REG_LINK_STATUS
`ethtool -S ethX`	`get_ethtool_stats`	tx/rx packets, bytes, drops, PFC counters
`ethtool --show-priv-flags ethX`	`get_priv_flags`	roce_mode, ib_mode flags

2.6 DCB Support via ethtool/dcbtool¶

For PFC/ETS/ECN configuration from the guest, the driver implements:

Interface	Callback
`dcbnl_rtnl_ops.ieee_getpfc`	Read PFC config from device
`dcbnl_rtnl_ops.ieee_setpfc`	Write PFC config to device
`dcbnl_rtnl_ops.ieee_getets`	Read ETS config
`dcbnl_rtnl_ops.ieee_setets`	Write ETS config

This allows standard mlnx_qos / lldptool / dcbtool commands to configure DCB.

3. Kernel RDMA Driver, `hicain_ib.ko`¶

3.1 Registration with ib_core¶

The RDMA driver registers a struct ib_device with the kernel's InfiniBand subsystem:

static const struct ib_device_ops hicain_ib_ops = {
    .owner               = THIS_MODULE,
    .driver_id            = RDMA_DRIVER_HICAIN,

    .query_device         = hicain_query_device,
    .query_port           = hicain_query_port,
    .query_gid            = hicain_query_gid,
    .query_pkey           = hicain_query_pkey,

    .alloc_ucontext       = hicain_alloc_ucontext,
    .dealloc_ucontext     = hicain_dealloc_ucontext,

    .alloc_pd             = hicain_alloc_pd,
    .dealloc_pd           = hicain_dealloc_pd,

    .create_qp            = hicain_create_qp,
    .modify_qp            = hicain_modify_qp,
    .destroy_qp           = hicain_destroy_qp,

    .create_cq            = hicain_create_cq,
    .destroy_cq           = hicain_destroy_cq,
    .poll_cq              = hicain_poll_cq,
    .req_notify_cq        = hicain_req_notify_cq,

    .reg_user_mr          = hicain_reg_user_mr,
    .dereg_mr             = hicain_dereg_mr,

    .post_send            = hicain_post_send,
    .post_recv            = hicain_post_recv,

    .get_port_immutable   = hicain_get_port_immutable,
};

3.2 RDMA Objects¶

The driver manages these kernel objects:

Object	Description
Protection Domain (PD)	Isolation boundary, simple counter, no hardware resource
Queue Pair (QP)	Send/receive queue, backed by kernel ring buffers
Completion Queue (CQ)	Completion notifications, ring buffer polled by userspace
Memory Region (MR)	Pinned user memory for DMA, uses `ib_umem_get()` to pin pages

3.3 Transport Modes¶

Mode	Protocol	How it Works
RoCEv2	UDP/IPv4 encapsulated RDMA	Driver builds ETH+IP+UDP+BTH headers, sends as Ethernet frame via `hicain_net`
IB	Native InfiniBand	Driver builds LRH+BTH headers, sends as IB frame via `hicain_net`

The mode is determined by the QP type and address resolution: - IBV_QPT_RC with GRH (Global Route Header) → RoCEv2 path - IBV_QPT_RC with LID-only addressing → IB path

3.4 Post Send / Post Recv Flow¶

Post Send (RDMA Write or Send):

1. Userspace calls ibv_post_send() → ioctl to kernel
2. hicain_post_send() picks up the work request
3. Build transport headers (RoCEv2: ETH+IP+UDP+BTH or IB: LRH+BTH)
4. Attach payload from registered MR
5. Submit as skb through hicain_net's ndo_start_xmit
6. On TX completion IRQ → generate CQ entry

Post Recv:

1. Userspace calls ibv_post_recv() → ioctl to kernel
2. hicain_post_recv() adds receive buffer to RX queue
3. On frame arrival (RX IRQ) → parse headers, match to QP
4. Copy payload to posted receive buffer
5. Generate CQ completion entry
6. Userspace polls CQ via ibv_poll_cq()

3.5 IB Tools Compatibility¶

With a properly registered ib_device, these tools work unmodified:

Tool	What it Does	Our Support
`ibstat`	Show IB device/port info	`query_device`, `query_port`
`ibv_devinfo`	Show verbs device capabilities	`query_device`
`ibv_devices`	List available RDMA devices	Device registration
`ibv_rc_pingpong`	RC send/recv test	Full QP/CQ/MR flow
`ib_send_bw`	Bandwidth benchmark	Full QP/CQ/MR flow
`ib_send_lat`	Latency benchmark	Full QP/CQ/MR flow
`rping`	RDMA CM ping	Requires RDMA CM support

4. Userspace Provider, `libhicain`¶

4.1 Purpose¶

libibverbs uses a plugin architecture. Each RDMA device needs a userspace provider library (.so) that implements the verbs_context_ops. This library is loaded by libibverbs when it detects our device.

4.2 Provider Registration¶

The provider registers via the standard rdma-core mechanism:

static const struct verbs_device_ops hicain_dev_ops = {
    .name = "hicain",
    .match_min_abi_version = HICAIN_ABI_VERSION,
    .match_max_abi_version = HICAIN_ABI_VERSION,
    .match_table = hicain_table,
    .alloc_device = hicain_device_alloc,
    .alloc_context = hicain_alloc_context,
};

4.3 Key Operations¶

Operation	Implementation
`alloc_context`	Open `/dev/infiniband/uverbsN`, mmap shared page
`alloc_pd`	Kernel ioctl to create PD
`create_qp`	Kernel ioctl, mmap send/recv ring buffers
`create_cq`	Kernel ioctl, mmap completion ring
`reg_mr`	Kernel ioctl to pin memory and get rkey/lkey
`post_send`	Write to send ring buffer (doorbell via mmap'd register)
`post_recv`	Write to recv ring buffer
`poll_cq`	Read from mmap'd completion ring (no syscall, fast path)

4.4 Integration with rdma-core¶

The provider .so is installed to the libibverbs provider directory:

/usr/lib/x86_64-linux-gnu/libibverbs/libhicain-rdmav34.so

Or via the RDMAV_PROVIDERS_PATH environment variable during development.

5. Directory Structure¶

vdc/
├── src/
│   ├── switch/                          ← Virtual switch daemon (done)
│   └── driver/
│       ├── Makefile                     ← Kernel module build (kbuild)
│       ├── hicain_net.c                 ← Network driver (PCI, net_device, ethtool)
│       ├── hicain_net.h                 ← Shared definitions (register offsets, etc.)
│       ├── hicain_ib.c                  ← RDMA driver (ib_device_ops)
│       ├── hicain_ib.h                  ← RDMA object definitions
│       └── userspace/
│           ├── Makefile                 ← Userspace provider build
│           ├── libhicain.c              ← libibverbs provider implementation
│           └── hicain-rdmav34.driver    ← Provider registration file
├── examples/
│   ├── roce_send_recv.c                 ← RoCEv2 RDMA Send/Recv example
│   ├── ib_send_recv.c                   ← InfiniBand RDMA Send/Recv example
│   ├── roce_rdma_write.c               ← RoCEv2 RDMA Write example
│   └── Makefile
├── tests/
│   ├── test_driver_load.py              ← Verify module loads, ethtool works
│   ├── test_ib_device.py                ← Verify ibstat/ibv_devinfo output
│   └── test_rdma_traffic.py             ← End-to-end RDMA send/recv between 2 VMs
└── docs/
    ├── HiCAIN_Driver_Design.md          ← This document
    └── HiCAIN_RoCE_IB_HowTo.md         ← User-facing guide

6. Build System¶

6.1 Kernel Modules¶

# Kbuild-style Makefile
obj-m += hicain_net.o
obj-m += hicain_ib.o

KDIR ?= /lib/modules/$(shell uname -r)/build

all:
    make -C $(KDIR) M=$(PWD) modules

clean:
    make -C $(KDIR) M=$(PWD) clean

install:
    make -C $(KDIR) M=$(PWD) modules_install
    depmod -a

6.2 Userspace Provider¶

CC = gcc
CFLAGS = -shared -fPIC -Wall -O2
LDFLAGS = -libverbs

libhicain.so: libhicain.c
    $(CC) $(CFLAGS) -o $@ $< $(LDFLAGS)

install: libhicain.so
    install -D libhicain.so /usr/lib/libibverbs/libhicain-rdmav34.so

7. Testing Strategy¶

7.1 Test Levels¶

Level	What	How
Module load	Driver loads, PCI probe succeeds, ethX appears	`insmod` + `ip link show`
ethtool	`ethtool -i`, `ethtool -S`, link status	Run ethtool, assert output
IB device	`ibstat`, `ibv_devinfo` show our device	Parse tool output
Loopback	Single-VM RDMA send to self	`ibv_rc_pingpong` loopback
Two-VM	RDMA between two VMs through virtual switch	`ibv_rc_pingpong` server/client

7.2 Test Environment¶

Tests require QEMU VMs with the RoCE-IB-vNIC device. For CI, we use the Python test framework:

def test_two_vm_rdma_pingpong():
    """
    1. Start virtual switch
    2. Start VM A with hicain-vnic on port 0
    3. Start VM B with hicain-vnic on port 1
    4. Inside VM A: ibv_rc_pingpong --server
    5. Inside VM B: ibv_rc_pingpong --client <VM_A_IP>
    6. Assert both sides report success
    """

8. Userspace Examples & HowTo¶

8.1 Example Programs¶

Example	Description
`roce_send_recv.c`	Two-process RoCEv2 Send/Recv using standard verbs API
`ib_send_recv.c`	Two-process IB Send/Recv using LID addressing
`roce_rdma_write.c`	One-sided RDMA Write (no remote CPU involvement)

Each example is a standalone C program that: - Uses only standard libibverbs API calls - Works with any RDMA device (not HiCAIN-specific) - Includes step-by-step output explaining each verbs call - Can run between two VMs in the HiCAIN lab

8.2 HowTo Guide Topics¶

The HowTo document (docs/HiCAIN_RoCE_IB_HowTo.md) will cover:

Loading kernel modules and verifying device presence
Configuring IP addresses for RoCEv2
Setting up PFC/ECN for lossless RoCEv2
Running ibv_rc_pingpong between two VMs
Running ib_send_bw bandwidth benchmarks
Writing your first RDMA application
Troubleshooting common issues

9. Resolved Design Decisions¶

Decision	Choice
RDMA CM support	Yes, in v1, implement RDMA CM for `rping`, `rdma_client/server`
Shared Receive Queue (SRQ)	No, basic RQ only for v1
Kernel upstream	Out-of-tree for v1, upstream candidate for v2 (RFC to LKML)
Provider ABI version	Latest stable (v34)