SuperMicro AOC-STGN-I2S labeled Coraid. Can't install drivers and use lan card

Question

Recently I purchased two lan cards SuperMicro AOC-STGN-I2S (Two 10Gb\s SFP+ ports). One of them work OK under Windows (my test PC) and under FreeBSD on my server. Second one is exactly the same card but with sticker on it "Coraid". The problem is that I can’t use it at all (in Windows and FreeBSD it is not working):

1. Bootloader gives me error during boot:

   PXE-E01: PCI Vendor and Device IDs do not much!
   

2. In Windows device manager I got two unknown devices without drivers and I can’t install official drivers from SuperMicro web site. The device ID is:

   PCI\VEN_8086&DEV_0001&CC_0200
   

   As I understand, Coraid label means that this lan card is flashed to another firmware to support network storage features that Coraid software provides. Is it possible to flash this card back to original SuperMicro firmware and use it as a usual network card?

Answer 1

I stumbled upon the same problem for my Intel X520-DA2. In order to solve it, I used an Ubuntu 18.04 server to alter the card's EEPROM content with ethtool.

modinfo shows the following for the ixgbe driver:

modinfo ixgbe.ko
filename:       /lib/modules/5.4.0-131-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        5.1.0-k
...

lspci shows what we got. It returns the two ports of the x520-DA2 as separate Ethernet cards:

lspci|grep Eth
01:00.0 Ethernet controller: Intel Corporation Device 0001 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Device 0001 (rev 01)

In order to query the card's vendor and subsystem IDs we call lspci again and append the PCI Bus ID from above. We can user either one of the IDs (e.g. 01:00.0).

lspci -vns 01:00.0|head -2
01:00.0 0200: 8086:0001 (rev 01)
        Subsystem: 1b52:10fb

"8086" is the vendor ID (for Intel). "0001" is the subvendor ID. It usually tells which device it is. In our case "0001" is definitely wrong. Since the subvendor ID is wrong, the driver module ixgbe does not feel responsible for the card. But we can still convice it otherwise. We simply load the module and then tell it to take care of our card. It is done like this:

modprobe ixgbe
echo 8086 0001 > /sys/bus/pci/drivers/ixgbe/new_id

Upon doing this, dmesg should show the following output:

dmesg
[  213.983160] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[  213.983161] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[  214.151320] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0
[  214.151607] ixgbe 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[  214.151946] ixgbe 0000:01:00.0: MAC: 2, PHY: 20, SFP+: 5, PBA No: G73129-006
[  214.151946] ixgbe 0000:01:00.0: a0:36:9f:xx:xx:xx
[  214.154111] ixgbe 0000:01:00.0 enp1s0f0: renamed from eth0
[  214.175253] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[  214.347439] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0
[  214.347730] ixgbe 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[  214.348051] ixgbe 0000:01:00.1: MAC: 2, PHY: 20, SFP+: 6, PBA No: G73129-006
[  214.348052] ixgbe 0000:01:00.1: a0:36:9f:xx:xx:xx
[  214.350172] ixgbe 0000:01:00.1 enp1s0f1: renamed from eth0
[  214.379294] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[  332.357800] ixgbe 0000:01:00.1: complete
[  332.381819] ixgbe 0000:01:00.0: complete

After we got the card recognized, there are two new Ethernet adapters present. We use ip to list them.

ip a

...
1: enp1s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether a0:36:9f:xx:xx:xx brd ff:ff:ff:ff:ff:ff
2: enp1s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether a0:36:9f:xx:xx:xx brd ff:ff:ff:ff:ff:ff

The cards usually shows up as enp1s0f0 and enp1s0f1. So lets have a look at the contents of the EEPROM using ethtool:

ethtool -e enp1s0f0

My EEPROM's content looks like this (excerpt):

0x0320:     03 00 1f 00 00 00 00 2b 03 53 fb 10 52 1b 01 00
0x0330:     01 00 f5 07 ed 10 35 00 64 6e 01 00 90 e2 ba 4c
0x0340:     8f dc e2 6a 03 00 05 00 3f 07 01 00 00 00 04 14
0x0350:     00 00 05 00 3f 17 01 00 00 00 04 14 00 00 07 00

According to @JustKoz the subvendor ID is at 0x034a & 0x034b as well as 0x0356 & 0x0357. Our subvendor ID "0001" is stored as "01 00" with bytes swapped.

0x0340:     xx xx xx xx xx xx xx xx xx xx 01 00 xx xx xx xx
0x0350:     xx xx xx xx xx xx 01 00 xx xx xx xx xx xx xx xx

The correct subvendor ID for the X520-DA2 is 10fb. In order to alter above addresses using ethtool, the commands to issue look like this:

ethtool -E enp1s0f0 magic 0x00018086 offset 0x034a value 0xfb
ethtool -E enp1s0f0 magic 0x00018086 offset 0x034b value 0x10
ethtool -E enp1s0f0 magic 0x00018086 offset 0x0356 value 0xfb
ethtool -E enp1s0f0 magic 0x00018086 offset 0x0357 value 0x10

I noticed that the values "1b52:10fb" shown for my card are also in the BIOS pic @JustKoz attached. This is the wrong subsystem ID.

The subsystem ID is stored at 0x032a-0x032d. The subsystem ID from the lscpi output can be found again here, but with bytes swapped and in wrong order.

0x0320:     xx xx xx xx xx xx xx xx xx xx fb 10 52 1b xx xx

Instead of 1b52:10fb we see "fb 10 52 1b"

Fixing the subsystem ID works the same way as fixing the subvendor ID. Since I did not have another card as refence for the IDs, I used this site. Because I rebooted to test if the card's subvendor ID was set corretly, we need to use 0x10fb8086 for magic in our ethtool call from now on.

ethtool -E enp1s0f0 magic 0x10fb8086 offset 0x032c value 0x86
ethtool -E enp1s0f0 magic 0x10fb8086 offset 0x032d value 0x80

ethtool -E enp1s0f0 magic 0x10fb8086 offset 0x032a value 0x02
ethtool -E enp1s0f0 magic 0x10fb8086 offset 0x032b value 0x00

The last two bytes of the 0x032x block had also been tempered with (I think), so I corrected those as well. I got the values from a Supermicro card as well as DELL variant of this card. The values are both identical on the other two cards, but it does not mean, that it is also the case for original Intel one. So take their content with a grain of salt. (Side note: The "Intel(R) PROSet Adapter Configuration Utility" under Windows checked the EEPROM and it came up green.)

ethtool -E enp1s0f0 magic 0x10fb8086 offset 0x032e value 0xa6
ethtool -E enp1s0f0 magic 0x10fb8086 offset 0x032f value 0x10

After applying all of the above changes I switched the card to a Windows machine. The driver installed without a problem and I see a 10Gbs link. Driver Status

To sum things up. This is the content of my card's EEPROM after fixing everything:

0x0320:     03 00 1f 00 00 00 00 2b 03 53 02 00 86 80 a6 10
0x0330:     01 00 f5 07 ed 10 35 00 64 6e 01 00 90 e2 ba 4c
0x0340:     8f dc e2 6a 03 00 05 00 3f 07 fb 10 00 00 04 14
0x0350:     00 00 05 00 3f 17 fb 10 00 00 04 14 00 00 07 00

Answer 2

I have at least 3 cards with the same problem, and probably a lot more. We have recently purchased about 20 of those cards to migrate a part of our production environment to 10G. 3 of the cards have a Coraid label on them, but a lot more of the cards are simply not recognized as a Supermicro AOC-STGN-I2S. Some of them are recognized correctly so I can easily identify them by testing them in a test system.

If you find a way to flash them, please let me know. I will do the same as soon as I have found something.

Jan Hugo Prins

Answer 3

I've contacted Supermicro and Coraid support. The answers are: 1. Supermicro gave me utility but it won't work with Coraid lan cards. 2. Coraid said they can help me only if I send them cards. They wanted 100$ per each card + shipping price. O_o Crazy guys.... So the easiest way is to throw Coraid card to the garbage and find cards with original Supermicro firmware.

Answer 4

I now know how to correct this but I can not figure out the magic key here are my attempts so far:

To gain access to the card under Ubuntu to the following:

sudo chmod 777 /sys/bus/pci/drivers/ixgbe/new_id
sudo echo 8086 0001 >/sys/bus/pci/drivers/ixgbe/new_id

Now you can use these commands to "verify" this table:

sudo ethtool -e enp1s0f0 offset 0x0340 length 0xb
sudo ethtool -e enp1s0f0 offset 0x0340 length 0xc
sudo ethtool -e enp1s0f0 offset 0x0350 length 0x7
sudo ethtool -e enp1s0f0 offset 0x0350 length 0x8


                                               __ __
>0x0340:         93 aa e2 6a 03 00 05 00 3f 06 fb 10 00 00 04 14
<0x0340:         c1 f6 e2 6a 03 00 05 00 3f 06 01 00 00 00 04 14
                                               b  c
                                   __ __
>0x0350:         00 00 05 00 3f 16 fb 10 00 00 04 14 00 00 07 00
<0x0350:         00 00 05 00 3f 16 01 00 00 00 04 14 00 00 07 00
                                   7  8

The problem is I can not figure out the magic value to overwrite the faulty Device ID:

sudo ethtool -E enp1s0f0 magic 0x00018086 offset 0340x0b value 0xfb
sudo ethtool -E enp1s0f0 magic 0x00018086 offset 0x0340 value bxfb
sudo ethtool -E enp1s0f0 magic 0x00018086 offset 0x0340 value 0bxfb

BIOS