2

I am trying to establish a 25 GbE link between Mikrotik CCR2004-1G-12S+2XS router and a server running Windows Server 2022 with Mellanox NIC (MCX512A-ADAT).

For cabling, I tried using the official Mikrotik DAC (XS+DA0003), unofficial FS.com with one side generic and one side "Mellanox-compatible", a pair of SFP28 from FS.com with SMF link and various other SFP/SFP+ modules just for testing.

Each time, the Mikrotik RouterOS shows a connected DAC/SFP incl. its serial number:

Mikrotik RouterOS screenshot

However, on the server (Mellanox) side, I only see information about the link being down or the cable unplugged:

Windows Server NIC adapter Control Panel screenshot

Mellanox NIC report

I tried changing FEC mode, rate (low/high), flow control and auto-negotiation on Mikrotik, but the link still did not work.

What am I doing wrong?

Edit:

I have also tried setting the FEC and link speed manually:

mlxlink -d mt4121_pciconf0.1 --link_mode_force --speeds 25G
mlxlink -d mt4121_pciconf0.1 -k FC --fec_speed 25G
mlxlink -d mt4121_pciconf0.1 -a TG

enter image description here

enter image description here

Vilican
  • 129
  • 1
  • 21
  • 1
    @PimpJuiceIT - I tried setting speed/FEC manually, but I am still getting the same problem. Firmware of the NIC/router are the latest available. – Vilican Jul 06 '23 at 12:12
  • 1
    Check out these tips just in case these help I wanted to share: https://www.excentis.com/blog/sfp-compatibility-issues-here-are-5-troubleshooting-tips/ I wish you good luck in getting this resolved. I don't have a lot of exposure to the SFP fast link configurations with Windows Server 2022. – Pimp Juice IT Jul 06 '23 at 18:32
  • Thanks for the resource. However, the connection does not work even with a brand-new DAC cable from FS, which has one end configured for Mikrotik and the other for that Mellanox NIC. Also, both are recognized by the systems, but the link is down for some reason. – Vilican Jul 06 '23 at 18:37
  • 1
    Getting close to needing a vendor/hardware manufacturer ticket to assist troubleshooting. Maybe rare but could be a defective cable. I suppose you don't have any known working DEC cable to test with but if so, perhaps that as another steps. I know it sounds weird but disabled and re-enabling the nic from the Windows side via the Windows GUI doesn't do anything. I look forward to hearing what the solution and problem was here regardless. – Pimp Juice IT Jul 06 '23 at 18:47
  • Actually, I do have several spare DACs or transceivers, but this link does not work with them either. I tried raising a ticket to vendor support, but they bounced me with a request to purchase their overpriced support plan. – Vilican Jul 06 '23 at 19:13
  • 1
    What's the hardware of the server you are working with like make and model? This may hold a clue potentially for what's listed as compatible so consider adding that to your question too. Perhaps it is a compatibility issue. If you have server hardware manufacturer support, maybe opening a ticket with them instead will help troubleshoot (or be a cheaper option to explore). For example: https://community.hpe.com/t5/proliant-servers-ml-dl-sl/mellanox-qsfp-breakout-cable-shows-disconnected-with-hpe-560-sfp/td-p/6994771 just as an example only, not specific to your hardware though – Pimp Juice IT Jul 06 '23 at 20:57
  • It's a custom-built server mostly from Supermicro parts. The MB is H13SSL-NT, which has this NIC officially supported. Not sure whether they want to support that, unfortunately. – Vilican Jul 09 '23 at 18:58
  • Start downgrading firmware then to a previous version, all the variations. Try to put in a few calls and see what you get for free call wise both vendors. BIOS firmware and the 25 gbps nic downgrade their firmware and see if you get lucky and find a compatible/bug free pair working nicely. – Pimp Juice IT Jul 09 '23 at 19:56

1 Answers1

0

Judging from your screenshots, both switch and NIC seem to be accepting their respective DAC module.

However, this section

enter image description here

seems to indicate a firmware problem with the NIC's PHY.

You should look for ways to verify and possibly update that firmware.

Zac67
  • 10,320
  • 2
  • 12
  • 32
  • I have tried re-flashing the NIC firmware using this manual: [https://network.nvidia.com/support/firmware/nic/](https://network.nvidia.com/support/firmware/nic/). However, it still does not work and reports the same message. – Vilican Jul 09 '23 at 19:04