HiCAIN RoCE & InfiniBand HowTo¶

A step-by-step guide for using the Cnuas Virtual AI HPC Infrastructure to run RDMA applications over RoCEv2 and InfiniBand.

Prerequisites¶

Linux kernel 6.19 (included as submodule)
QEMU with HiCAIN RoCE-IB-vNIC (included as submodule)
rdma-core (libibverbs, ibverbs-utils, perftest)
HiCAIN virtual switch built and running

1. Build Everything¶

./build_and_test.sh

cd src/hinic/kernel
make KDIR=../../linux

cd userspace
make
sudo make install

2. Start the Virtual Switch¶

cd src/hiswitch/switch/build
./hicain-vswitchd --run-dir /tmp/hicain --foreground --log-level info

3. Launch Two VMs with RoCE-IB-vNICs¶

Terminal A (VM 1):

qemu-system-x86_64 \
    -m 2G -smp 2 \
    -drive file=vm1.qcow2,format=qcow2 \
    -device hicain-vnic,socket_path=/tmp/hicain/port_0.sock \
    -nographic

Terminal B (VM 2):

qemu-system-x86_64 \
    -m 2G -smp 2 \
    -drive file=vm2.qcow2,format=qcow2 \
    -device hicain-vnic,socket_path=/tmp/hicain/port_1.sock \
    -nographic

4. Load Kernel Modules (Inside Each VM)¶

insmod hicain_net.ko
insmod hicain_ib.ko

Verify:

ip link show
ethtool -i eth1
ibv_devinfo
ibstat

Expected output from ibv_devinfo:

hca_id: hicain_ib0
    transport:          InfiniBand (0)
    fw_ver:             0.1.0
    node_guid:          0248:4341:4900:0001
    sys_image_guid:     0248:4341:4900:0001
    vendor_id:          0x1ed5
    vendor_part_id:     0xca10
    hw_ver:             0x1
    phys_port_cnt:      1
        port:   1
            state:          PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:     4096 (5)
            sm_lid:         1
            port_lid:       1
            port_lmc:       0x00
            link_layer:     InfiniBand

5. Configure IP Addresses (RoCEv2)¶

Inside VM 1:

ip addr add 10.0.0.1/24 dev eth1
ip link set eth1 up

Inside VM 2:

ip addr add 10.0.0.2/24 dev eth1
ip link set eth1 up

Verify connectivity:

ping 10.0.0.2

6. Configure DCB for Lossless RoCEv2¶

Inside each VM:

# Enable PFC on priority 3 (RDMA traffic class)
mlnx_qos -i eth1 --pfc 0,0,0,1,0,0,0,0

# Or via the switch CLI from the host:
hicain-cli pfc enable 0 --priorities 3
hicain-cli ecn set 0 --enable --threshold 32768 --probability 10

7. Run RDMA Ping-Pong (RoCEv2)¶

VM 1 (server):

ibv_rc_pingpong -d hicain_ib0 -g 0

VM 2 (client):

ibv_rc_pingpong -d hicain_ib0 -g 0 10.0.0.1

Expected output:

  local address:  LID 0x0001, QPN 0x000001, PSN 0x000000, GID fe80::248:4341:4900:1
  remote address: LID 0x0001, QPN 0x000001, PSN 0x000000, GID fe80::248:4341:4900:2
  8192000 bytes in 0.05 seconds = 1310.72 Mbit/sec
  1000 iters in 0.05 seconds = 50.00 usec/iter

8. Run RDMA Bandwidth Test¶

VM 1 (server):

ib_send_bw -d hicain_ib0 --report_gbits

VM 2 (client):

ib_send_bw -d hicain_ib0 --report_gbits 10.0.0.1

9. Run InfiniBand (LID-Based) Test¶

Configure LID routing on the switch:

hicain-cli port set-mode 0 STRICT_IB
hicain-cli port set-mode 1 STRICT_IB
hicain-cli lft add --dlid 1 --port 0
hicain-cli lft add --dlid 2 --port 1

VM 1 (server):

ibv_rc_pingpong -d hicain_ib0

VM 2 (client):

ibv_rc_pingpong -d hicain_ib0 -l 1

10. Run RDMA CM Test (rping)¶

VM 1 (server):

rping -s -v -a 10.0.0.1

VM 2 (client):

rping -c -v -a 10.0.0.1

11. Run the Example Programs¶

Build and run the included examples:

cd examples
make

# RoCEv2 mode
# Terminal 1 (server): ./roce_send_recv
# Terminal 2 (client): ./roce_send_recv 10.0.0.1

# InfiniBand mode
# Terminal 1 (server): ./ib_send_recv
# Terminal 2 (client): ./ib_send_recv <server_lid>

12. Monitoring with ethtool¶

ethtool -i eth1
ethtool eth1
ethtool -S eth1

Example ethtool -S output:

NIC statistics:
     tx_packets: 15432
     tx_bytes: 4523100
     rx_packets: 15100
     rx_bytes: 4418000
     tx_drops: 0
     rx_drops: 3

13. Troubleshooting¶

Symptom	Check
No RDMA device found	`lsmod \\| grep hicain`, modules loaded?
Port state DOWN	`ethtool eth1`, is the vNIC connected to the switch socket?
ibv_rc_pingpong hangs	Switch ports in correct mode? LFT entries configured?
PFC not working	`hicain-cli pfc status 0`, PFC enabled on correct priority?
High drops	`ethtool -S eth1`, check rx_drops, tx_drops counters
Cannot ping	`ip addr show eth1`, IP assigned? Switch port in ETH mode?