Skip to content

HiCAIN HiLink Switch Design

1. Overview

HiLink is the inter-GPU fabric in the Cnuas, mirroring NVIDIA NVLink and NVSwitch. The HiLink Switch is a separate daemon from the HiCAIN Switch (which carries RoCE/IB traffic) so that students see the same topological separation as in real datacentres: NIC fabric and GPU fabric are independent.

Real World HiCAIN
NVLink HiLink (per-GPU endpoint protocol)
NVSwitch HiLink Switch (higpu-link-switchd)
nvidia-fabricmanager hi-fabricmgr

2. Design Goals

  1. Independent of the HiCAIN Switch, separate process, separate visualisation
  2. Same 10-port chassis convention as the HiCAIN Switch (consistency)
  3. Hybrid datapath: ivshmem for same-host GPU-to-GPU pairs (mimics NVLink bandwidth), UNIX socket through the switch for everything else
  4. SIMT-aware routing, collective primitives (AllReduce, AllGather) can be accelerated by switch-resident reduction (mirrors NVSwitch SHARP)
  5. Ports are GPU-facing, no Ethernet or InfiniBand classification logic

3. Identifiers

Field Value
Daemon higpu-link-switchd
Run dir default /var/run/hilink
Mgmt socket /var/run/hilink/mgmt.sock
Port sockets /var/run/hilink/port_0.sock through port_9.sock
Wire protocol HiLink (raw frames over SOCK_SEQPACKET)

4. Port Layout

Port range Role Notes
0, 7 Fabric (GPU connections) Each port serves one HiGPU endpoint
8 Inter-switch link Connect to peer HiLink Switch in second rack (when present)
9 Console / OTel Telemetry export, no data plane

Same role concept as the HiCAIN Switch, different protocol on the wire.

5.1 Frame Format

flowchart LR M["Magic<br/>2 B"] --> V["Ver<br/>1 B"] --> T["Type<br/>1 B"] --> S["Src GPU<br/>2 B"] --> D["Dst GPU<br/>2 B"] --> L["Length<br/>4 B"] --> P["Payload<br/>N B"] classDef hdr fill:#dbeafe,stroke:#1e40af,stroke-width:2px,color:#1e3a8a; classDef pay fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f; class M,V,T,S,D,L hdr; class P pay;
Field Size Value
Magic 2 B 0x484C ("HL")
Version 1 B 0x01
Type 1 B See type table below
Src GPU ID 2 B Sender GPU global ID
Dst GPU ID 2 B Receiver GPU global ID (or 0xFFFF broadcast)
Length 4 B Payload bytes
Payload N B Type-dependent

5.2 Frame Types

Code Type Payload
0x01 DATA Raw bytes (used by HiSHMEM put/get and HiCCL stages)
0x02 DOORBELL 64-bit doorbell value
0x03 ATOMIC Atomic op descriptor (op, addr, value)
0x04 COLLECTIVE_REDUCE Reduction stage (op, dtype, count, partial)
0x05 COLLECTIVE_GATHER Gather stage
0x06 DISCOVERY GPU announces itself on link bring-up
0x07 LINK_KEEPALIVE Periodic heartbeat

5.3 GPU IDs

Each HiGPU has a globally unique 16-bit ID derived from (rack_id << 8) | slot_id. The HiLink Switch keeps a forwarding table from GPU ID to port, analogous to the L2 FDB in the HiCAIN Switch.

6. Datapath

6.1 Hybrid Transport

Two transports, chosen automatically by the GPU driver based on peer location:

Path Transport Used When
Fast path ivshmem (POSIX shared memory + UNIX socket for signalling) Two HiGPUs are in QEMU instances on the same host
Switch path UNIX SOCK_SEQPACKET to HiLink Switch Cross-host or multi-hop

The HiLink Switch maintains a peer registry. GPU driver queries the switch on hi_link_open() to learn whether a direct ivshmem channel exists for a given peer.

6.2 ivshmem Fast Path

When two HiGPUs are on the same host:

  1. QEMU launched with -object memory-backend-file,id=hilink-AB,... shared between the two QEMU processes
  2. Each HiGPU sees the shared region in BAR1 of the peer (or a dedicated BAR3 for inter-GPU memory)
  3. Direct loads/stores to peer memory; doorbells go through a QEMU-instantiated ivshmem-doorbell device
  4. No frame encapsulation, raw memory access

This is roughly how NVLink P2P works on real hardware.

6.3 Switch Path

When direct ivshmem isn't available:

  1. GPU driver writes HiLink frame to its endpoint TX queue
  2. QEMU emulator picks up the frame, sends over UNIX socket to switch
  3. Switch routes by Dst GPU ID, forwards to destination port
  4. Destination QEMU receives frame, writes payload to GPU's RX buffer
  5. RX MSI raised in destination guest

Higher latency than ivshmem but works across racks and hosts.

6.4 Collective Acceleration

The switch can fold reductions in flight:

  1. HiCCL launches AllReduce(sum, X)
  2. Each GPU sends COLLECTIVE_REDUCE to switch
  3. Switch sums incoming partials in real time
  4. When all GPUs have contributed, switch broadcasts the sum back
  5. This mirrors NVSwitch SHARP

This is a v2 feature; v1 implements collectives as N-stage point-to-point.

7. Switch Daemon

7.1 Architecture

Same single-threaded epoll model as the HiCAIN Switch. Reuses the same patterns:

  • SOCK_SEQPACKET listeners on ports 0, 8
  • SOCK_STREAM listener on management socket
  • Single-threaded event loop

7.2 Forwarding Tables

Table Key Value
GPU FDB 16-bit GPU ID Egress port
Group table Group ID List of GPU IDs (used by collectives)

GPU FDB populated automatically via DISCOVERY frames on link up.

7.3 Management API

JSON over UNIX socket, same pattern as the HiCAIN Switch:

Command Effect
port_status Per-port state, counters, peer GPU ID
gpu_fdb_dump List of (GPU ID → port) mappings
group_create Create a collective group
group_dump List groups
set_link_state Up / down a port
telemetry_dump Frame counters per port and per type
telemetry_clear Reset counters

8. WebUI Visualisation

The Cnuas rack view gains a second top-of-rack switch slot:

flowchart TB S1["HiCAIN Switch (RoCE/IB) &nbsp;&mdash;&nbsp; ports 0-7 to NICs"] S2["HiLink Switch &nbsp;&mdash;&nbsp; ports 0-7 to HiGPU endpoints"] C0["Sled chassis 0 &nbsp;&mdash;&nbsp; VMs, each VM may host one HiGPU"] C1["Sled chassis 1"] PSU["PSU"] BBU["BBU"] S1 --- S2 --- C0 --- C1 --- PSU --- BBU classDef sw fill:#dbeafe,stroke:#1e40af,stroke-width:2px,color:#1e3a8a; classDef link fill:#e0e7ff,stroke:#3730a3,stroke-width:2px,color:#1e1b4b; classDef sled fill:#dcfce7,stroke:#166534,stroke-width:2px,color:#14532d; classDef pwr fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f; class S1 sw; class S2 link; class C0,C1 sled; class PSU,BBU pwr;

The HiLink Switch chassis is rendered with the same visual style as the HiCAIN Switch (port LEDs, vendor strip) but with a different colour accent to distinguish it.

9. CLI

hilink-cli, analogous to hicain-cli:

hilink-cli port status
hilink-cli port status 0
hilink-cli gpu-fdb show
hilink-cli group create --name allred-group --gpus 0,1,2,3
hilink-cli telemetry show

Reuses the same Typer + Rich + Pydantic stack as hicain-cli.

10. Telemetry

Per-port counters:

Counter
rx_frames
tx_frames
rx_bytes
tx_bytes
rx_drops
tx_drops
collective_reduce_count
collective_gather_count
latency_us_p50
latency_us_p99

Exported via OTel on port 9 (same pattern as HiCAIN Switch).

11. Phasing

Phase Scope
1 Daemon scaffold, port sockets, mgmt socket, JSON API
2 DATA frame forwarding by Dst GPU ID, FDB learning via DISCOVERY
3 DOORBELL and ATOMIC frame types
4 hilink-cli
5 ivshmem fast-path: switch advertises same-host peers
6 WebUI rack rendering with HiLink Switch slot
7 COLLECTIVE_REDUCE, COLLECTIVE_GATHER stages (v1: passthrough)
8 OTel telemetry export
9 Switch-side collective folding (SHARP-style), v2

12. Relationship to HiCAIN Switch

Aspect HiCAIN Switch HiLink Switch
Process hicain-vswitchd higpu-link-switchd
Wire protocol RoCEv2 / InfiniBand frames HiLink frames
Endpoint device RoCE-IB-vNIC (hicain-vnic) HiGPU (higpu)
Routing key MAC / LID GPU ID
Use case NIC traffic, storage, SSH, HiCCL fallback GPU-to-GPU compute traffic
Run dir /var/run/hicain /var/run/hilink

The two switches are completely independent processes. A typical rack runs both side by side. HiCCL can fall back to using RoCE through the HiCAIN Switch if HiLink is unavailable, but that is a software policy decision in HiCCL, not a coupling between the daemons.

13. Directory Structure (planned)

src/
  hilink-switch/             # daemon source
    src/
    include/
    BUILD.bazel
  hilink-cli/                # CLI
    hilink_cli/
    BUILD.bazel
docs/
  HiCAIN_HiLink_Switch_Design.md

14. Open Questions

  1. GPU ID assignment: static at QEMU launch via property, or learned dynamically from a discovery handshake?
  2. ivshmem device naming: one shared region per pair (n^2 regions), or one big region and a routing layer? Pair-wise scales worse but is simpler.
  3. Frame size: 9 KB jumbo (matching HiCAIN Switch) or larger (e.g. 64 KB to match NVLink burst behaviour)?
  4. Authentication: do we need any peer authentication on link bring-up, or trust the local socket namespace?