vSphere 6.7 iSER setup with Mellanox ConnectX-3
I wanted to see if enabling iSER for my ESXi servers would improve the performance of iSCSI, since RDMA should offload a lot of cpu cycles.
Unfortunately it is not just a matter of checking a checkbox in ESXi to get it enabled and working, and even if you do all the required steps - it might just not work anyway.
For my Melanox ConnectX-3 (non pro) I had to this on my ESXiinstallation(s)
#enable iser module esxcli system module set --enabled=true --module=iser #adds the rdma iser adapter - without this you will not have an iser adapter to configure esxcli rdma iser add esxcli system module parameters set -m nmlx4_en -p "pfctx=0x08 pfcrx=0x08" esxcli system module parameters set -m nmlx4_core -p "enable_qos=1 enable_rocev2=0" esxcli system module parameters set -m nmlx4_rdma -p "pcp_force=3 dscp_force=24 dscp_to_pcp=0" #tag the vmkernel adapter with VSAN, exchange vmk0 for whatever interface(s) you use for iscsi esxcli network ip interface tag add -i vmk0 -t VSAN
That should be the setup on the ESXiside - furthermore i had to ensure the the Mellanox cards were set up correctly for iSER - according to Mellanox iSER is not supported if you have SRIOV enabled, so to disable that on the cards itself I had to first install Mellanox’s management software on my ESXi’s.
cd /tmp wget https://www.mellanox.com/downloads/MFT/vmware_6.5_native/nmst-22.214.171.124-1OEM.6126.96.36.19998673.x86_64.vib wget https://www.mellanox.com/downloads/MFT/vmware_6.5_native/mft-188.8.131.52-10EM-6184.108.40.20698673.x86_64.vib esxcli software vib install -v /tmp/mft-220.127.116.11-10EM-618.104.22.16898673.x86_64.vib --no-sig-check esxcli software vib install -v /tmp/nmst-22.214.171.124-1OEM.6126.96.36.19998673.x86_64.vib --no-sig-check
Notice the –no-sig-check - this is required since the certificate that these were signed with are too old
Reboot the server and then the mellanox tools are located in /opt/bin/mellanox
Configure the cards to not use SRIOV - exchange 02:00.0 for the actualy address where the mellanox card is located - you can see this in the hardware list on your ESXi server.
/opt/mellanox/bin/mlxconfig -d 02:00.0 s SRIOV_EN=0 NUM_OF_VFS=0 LINK_TYPE_P1=4 LINK_TYPE_P2=4
Reboot the server
After that it should be as simple as adding a dynamic target for the iSER iSCSI adapters and remember to also add a vmkernel port binding
If you are lucky and your switch/SAN devices is configured correctly then you should see storage devices that are the iSER targets.