Mellanox Connect x4 SRIOV Ubuntu 22.04

SRIOV-CNI support Mellanox ConnectX®-4 Lx and ConnectX®-5 adapter cards. To enable SR-IOV functionality the following steps are required:
1.Verify that the system has a Mellanox network adapter (HCA/NIC) installed.

# lspci -v | grep Mellanox
58:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT
58:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT

1- Enable SR-IOV in the NIC’s Firmware.

Installing Mellanox Management Tools (MFT) or mstflint is a pre-requisite, MFT can be downloaded from http://www.mellanox.com/page/management_tools

Download the mft package from https://www.mellanox.com/downloads/MFT/mft-4.29.0-131-x86_64-deb.tgz

install dependency package

sudo apt install gcc make dkms

extract package

tar -xzvf mft-4.29.0-131-x86_64-deb.tgz

go to folder we’ve extracted and execute install.sh

#cd mft-4.26.1-6-x86_64-deb
#./install.sh

Use Mellanox Firmware Tools package to enable and configure SR-IOV in firmware

# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices

Enable SR-IOV NIC’s Driver

# mstconfig -d 58:00.0 set SRIOV_EN=1 NUM_OF_VFS=8
...
Apply new Configuration ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

Where 58:00.0 is the NIC’s PCI address.
Reboot the machine

to Create VF interface use command
#echo 4 > /sys/class/net/enp2s0f0/device/sriov_numvfs
check vf interface

# lspci | grep Mellanox
58:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
58:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
58:00.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
58:00.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
58:00.4 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
58:00.5 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

Trust The VF interface use command:
#ip link set dev ens4f0np0 vf 0 trust on

To change the number of VFs reset the number to 0 then set the needed number

echo 0 > /sys/class/net/enp2s0f0/device/sriov_numvfs

When machine reboot VF interface cant load automaticaly to load automatically we must create a service and run when machine booting

Create Script on /usr/bin/vfinterface.sh
# nano /usr/bin/vfinterface.sh

#!/bin/bash
echo 2 > /sys/class/net/ens4f0np0/device/sriov_numvfs
ip link set dev ens4f0np0 vf 0 trust on
ip link set dev ens4f0np0 vf 1 trust on

and create service for load that script on boot
# nano /etc/systemd/system/vfinterface.service

[Unit]
Description=VF Interface

[Service]
ExecStart=/usr/bin/vfinterface.sh
User=root
Group=root

[Install]
WantedBy=multi-user.target

test the service
#systemctl start vfinterface.service

load service automatically every booting
#systemctl enable vfinterface.service