Install Mellanox Connect X SRIOV Openstack Xena, Using Kolla

Version

[Openstack Xena, Ubuntu 20.04]
eno1

eno1.888 : Internal
eno1.2005 : External

Reference

SRIOV Mellanox : https://support.mellanox.com/s/article/HowTo-Configure-SR-IOV-for-ConnectX-4-ConnectX-5-ConnectX-6-with-KVM-Ethernet
Advance Config Kolla : https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html
ML2 Config Template : https://wiki.openstack.org/wiki/Ml2_conf.ini_File
SRIOV Openstack : https://docs.openstack.org/neutron/latest/admin/config-sriov.html
SRIOV Kolla : https://docs.openstack.org/kolla-ansible/latest/reference/networking/sriov.html

Installation

Install NIC ConnectX 4 or 5 or 6

#lspci | grep Ethernet
2f:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 2f:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

On BIOS Enable “SR-IOV” and “VT-D”/”Intel Virtualization Technology” in the system BIOS

Update Grub for “iommu” parameter

$sudo nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT=”intel_iommu=on”
$sudo update-grub

Install the latest MLNX_OFED driver on the server and on the VM

Download OFED Linux-Ubuntu = https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

Download > 5.6-2.0.9.0 > Ubuntu > Ubuntu 20.04 > X86_64 > MLNX_OFED_LINUX-5.6-2.0.9.0-ubuntu20.04-x86_64.tgz

->Put in SRIOV-Compute

$ tar -xfv MLNX_OFED_LINUX-5.6-2.0.9.0-ubuntu20.04-x86_64.tgz
$cd MLNX_OFED_LINUX-5.6-2.0.9.0-ubuntu20.04-x86_64
sudo ./mlnxofedinstall
->Y
*this will do install for all package, if for some reason it failed, try to install basic package only. You can check this link https://docs.nvidia.com/networking/display/MLNXOFEDv461000/Installing+Mellanox+OFED

Enable SRIOV on Firmware

#mst start

Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices

Locate the HCA device on the desired PCI slot.

#mst status

MST modules:

…

MST devices:/dev/mst/mt4119_pciconf0 - PCI configuration cycles access. ……

Query the Status of the device

#mlxconfig -d /dev/mst/mt4119_pciconf0 q

Device #1:
Device type: ConnectX4 PCI device: /dev/mst/mt4115_pciconf0 …. Configurations: Current SRIOV_EN 0 NUM_OF_VFS 0 ….

Enable SR-IOV , set the desired number of VFs.

SRIOV_EN=1 NUM_OF_VFS=4 ; This is an example with 4 VFs
mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4

Note: At this point, the VFs are not seen via lspci. Only when SR-IOV is enabled on the MLNX_OFED driver, you will be able to see them.

Reboot

#reboot

Enable SR-IOV on the MLNX_OFED Driver, Find the device, In this case, we have mlx5_1 active and up on interface sriov ens2f0np0

ibstat -> Check Status with ibdev2netdev
mlx5_0 port 1 ==> ens2f0np0 (Up)
mlx5_1 port 1 ==> ens2f1np1 (Down)
**If the state is down, check if the cable/connection status,,, if it is OK try restart openibd service,
#service openibd restart

also bring up the interface
#ifup ens2f0np0

Get the total VFs that are allowed and configured in the firmware.

cat /sys/class/net/[Physical Interface ie ens2f0np0]/device/sriov_totalvfs
4

Get the current number of VFs on this device:

There are several ways to do so:
# cat /sys/class/infiniband/mlx5_1/device/mlx5_num_vfs
0
#cat /sys/class/net/ens2f0np0/device/sriov_numvfs
0
#cat /sys/class/net/ens2f1np1/device/mlx5_num_vfs
0

Set the desired number of VFs:

The number of VFs can be set via two parameters, depends on the kernel version.
There are several ways to do so:

option 1 (example create 4 VF to num_vfs):

echo 4 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

cat /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

option 2:

echo 4 > /sys/class/net/ens2f0np0/device/sriov_numvfs

cat /sys/class/net/ens2f0np0/device/sriov_numvfs

option 3:

echo 4 > /sys/class/net/ens2f0np0/device/mlx5_num_vfs

Note : Changing the number of VFs is non-persistent and does not survive a server reboot!

By now, the VF Shold be appear, check the PCI bus:

#lspci -D | grep Mellanox

0000:2f:00.0 Ethernet controller:

Mellanox Technologies MT27800 Family [ConnectX-5]
0000:2f:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
0000:2f:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:2f:00.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:2f:00.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:2f:00.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:2f:00.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:2f:00.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:2f:01.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:2f:01.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] ibdev2netdev -v 0000:2f:00.0 mlx5_0 (MT4119 – MCX516A-CCAT) CX516A – ConnectX-5 QSFP28 fw 16.33.1048 port 1 (ACTIVE) ==> ens2f0np0 (Up)
0000:2f:00.1 mlx5_1 (MT4119 – MCX516A-CCAT) CX516A – ConnectX-5 QSFP28 fw 16.33.1048 port 1 (DOWN ) ==> ens2f1np1 (Down)
0000:2f:00.6 mlx5_2 (MT4120 – NA) fw 16.33.1048 port 1 (DOWN ) ==> ens2f1np0v0 (Down)
0000:2f:00.7 mlx5_3 (MT4120 – NA) fw 16.33.1048 port 1 (DOWN ) ==> ens2f1np0v1 (Down)
0000:2f:01.0 mlx5_4 (MT4120 – NA) fw 16.33.1048 port 1 (DOWN ) ==> ens2f1np0v2 (Down)
0000:2f:01.1 mlx5_5 (MT4120 – NA) fw 16.33.1048 port 1 (DOWN ) ==> ens2f1np0v3 (Down)
0000:2f:00.2 mlx5_6 (MT4120 – NA) fw 16.33.1048 port 1 (DOWN ) ==> ens2f0np0v0 (Down)
0000:2f:00.3 mlx5_7 (MT4120 – NA) fw 16.33.1048 port 1 (DOWN ) ==> ens2f0np0v1 (Down)
0000:2f:00.4 mlx5_8 (MT4120 – NA) fw 16.33.1048 port 1 (DOWN ) ==> ens2f0np0v2 (Down)

Enable SRIOV in Openstack via Kolla-ansible.

#nano /etc/kolla/globals.yml
enable_neutron_sriov: “yes”
neutron_sriov_physnet_mappings:
sriovtenant1: ens2f0np0

Add folder and file kolla custom config for ml2_conf

#nano /etc/kolla/config/neutron/ml2_conf.ini
[ml2_type_vlan]
network_vlan_ranges = physnet1:2000:3000,sriovtenant1:1000:1999
[ml2_type_flat]
flat_networks = sriovtenant1

Add folder and file kolla custom config for nova.conf

#nano /etc/kolla/config/nova.conf
[filter_scheduler]
enabled_filters = PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters

Kolla deploy or reconfigure

kolla-ansible -i multinode reconfigure

Verify

ip -d link show ens2f0np0

4: ens2f0np0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b8:59:9f:1d:05:aa brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9978 addrgenmode eui64 numtxqueues 1152 numrxqueues 96 gso_max_size 65536 gso_max_segs 65535 portname p0 switchid aa051d00039f59b8
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

docker ps –filter name=neutron_sriov_agent

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5d728558d2e4 quay.io/openstack.kolla/ubuntu-source-neutron-sriov-agent:yoga “dumb-init –single-…” 46 minutes ago Up 45 minutes (healthy) neutron_sriov_agent

Test Openstack Operations

Operation : Create Network SRIOV

openstack network create –project=service \
–provider-network-type=vlan \
–provider-physical-network=sriovtenant1 \
–provider-segment=110 \
sriovnet1

Operation : Create Subnet

openstack subnet create –network=sriovnet1 \
–subnet-range=10.1.0.0/24 \
–allocation-pool start=10.1.0.200,end=10.1.0.210 \
sriovnet1_sub1

Operation : Create port
$ openstack port create –network sriovnet1 –vnic-type direct sriovport

Operation : Start a new instance with the SRIOV port assigned:
$ openstack server create –flavor m1.tiny \
–image cirros \
–nic port-id=`openstack port list | grep sriovport | awk '{print $2}'` \
vm-sriov

After creating a network, we can see the VLAN/Provider on the VF Link

$ip -d link show ens2f0np0

$ ip -d link show ens2f0np0
21: ens2f0np0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b8:59:9f:1d:05:aa brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9978 addrgenmode eui64 numtxqueues 1024 numrxqueues 64 gso_max_size 65536 gso_max_segs 65535 portname p0 switchid aa051d00039f59b8
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether fa:16:3e:af:4c:d1 brd ff:ff:ff:ff:ff:ff, vlan 110, spoof checking on, link-state enable, trust off, query_rss off
vf 2 link/ether fa:16:3e:de:a8:14 brd ff:ff:ff:ff:ff:ff, vlan 103, spoof checking on, link-state enable, trust off, query_rss off
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off