Background

Recently, I was configuring a network for the server room and came across a requirement to use ConnectX-4 as an Ethernet card, which supports both Infiniband and Ethernet, but the default is Infiniband mode, so I need to use the mlxconfig tool to do this switch.

How to switch

In the Using mlxconfig documentation, it is written how to switch the NIC to Infiniband mode.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$ mlxconfig -d /dev/mst/mt4103_pci_cr0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1
 
Device #1:
----------
Device type:   ConnectX3Pro
PCI device:    /dev/mst/mt4103_pci_cr0
Configurations:        Next Boot        New
  LINK_TYPE_P1         ETH(2)           IB(1)
  LINK_TYPE_P2         ETH(2)           IB(1)
 
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

Then, we just need to do the opposite and set the mode to ETH(2) and that’s it.

MST Installation

To use mlxconfig, you need to install MFT(Mellanox Firmware Tools). We are using Debian bookworm, so we have to download DEB.

1
2
3

wget https://www.mellanox.com/downloads/MFT/mft-4.20.1-14-x86_64-deb.tgz
unar mft-4.20.1-14-x86_64-deb.tgz
cd mft-4.20.1-14-x86_64-deb

Tried to install it with sudo . /install.sh to install, and found that dkms was reporting errors. Checking the logs, I found that because the kernel is too high (5.18), a function has modified its usage, i.e. to change the pci_unmap_single call to dma_unmap_single, and to change the first parameter, e.g. linux commit a2e759612e5ff3858856fe97be5245eecb84e29b points out.

1
2

-           pci_unmap_single(dev->pci_dev, dev->dma_props[i].dma_map, DMA_MBOX_SIZE, DMA_BIDIRECTIONAL);
+           dma_unmap_single(&dev->pci_dev->dev, dev->dma_props[i].dma_map, DMA_MBOX_SIZE, DMA_BIDIRECTIONAL);

After the changes, manually sudo dkms install kernel-mft-dkms/4.20.1 and it compiles successfully. Install mft manually again and start the service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

$ sudo dpkg -i DEBS/mft_4.20.1-14_amd64.deb
$ sudo mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
[warn] mst_pciconf is already loaded, skipping
Create devices
Unloading MST PCI module (unused) - Success
$ sudo mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mtxxxx_pciconf0         - PCI configuration cycles access.
                                   domainšŸšŒdev.fn=0000:xx:xx.0 addr.reg=yy data.reg=zz cr_bar.gw_offset=-1
                                   Chip revision is: 00

Now that it’s installed, the final execution of mlxconfig will switch to Ethernet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ sudo mlxconfig -d /dev/mst/mtxxxx_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Device #1:
----------

Device type:    ConnectX4
Name:           REDACTED
Description:    ConnectX-4 VPI adapter card; FDR IB (56Gb/s) and 40GbE; dual-port QSFP28; PCIe3.0 x8; ROHS R6
Device:         /dev/mst/mtxxxx_pciconf0

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        IB(1)           ETH(2)
         LINK_TYPE_P2                        IB(1)           ETH(2)

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.