0

I'm trying to write a kernel module for an Intel FPGA design supporting PCIe SR-IOV and placed in the x16 PCIe slot of an IBase M991 Mainboard (Q170 PCH, VT-d activated in BIOS, Integrated graphics only mode enabled).
The CPU is an Intel Core i7-6700TE, which also supports virtualization.
Furthermore I'm using a Yocto - Morty Distribution (Linux Kernel 4.19) with the following Kconfigs enabled:

CONFIG_PCI_IOV=y
CONFIG_PCI_DEBUG=y
CONFIG_INTEL_IOMMU_SVM=y
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
CONFIG_INTEL_IOMMU_DEFAULT_ON=y
CONFIG_IRQ_REMAP=y
CONFIG_IOMMU_DEFAULT_PASSTHROUGH=y
CONFIG_DYNAMIC_DEBUG=y

When doing all of this I see my driver loading (probe function gets called), but after calling pci_enable_sriov with the number of VF I want to activate I get the kernel message
not enough MMIO resources for SR-IOV
What am I doing wrong here? Is there an init function I need to call?
Many thanks for your help.

Edit: More information about the PCIe device:
1 PF, 8 VF
2 BARs (BAR0 and BAR2)
non prefetchable, 32 bit BARs
each BAR size is 4 kB (12bit)

Krustenkaese
  • 155
  • 1
  • 9
  • The MMIO address range behind each root port and bridge is set up by the BIOS. I’m not sure, but maybe there is a BIOS setting to get it to block off enough MMIO space for the device and all of its VFs? Or maybe an “SR-IOV Enable” that tells it to look for the number of VFs supported by the device. – prl Apr 23 '21 at 09:23
  • When I search for that message I get lots of hits. Are none of them helpful to you? – prl Apr 23 '21 at 09:28
  • Sadly yes. The Kernelconfigs already take care of the provided kernel cmdline arguments like pci=realloc or intel_iommu=on. I also added them manually, but behaviour didn't change – Krustenkaese Apr 23 '21 at 10:28
  • @prl, PCIe is flexible in that sense. How by your opinion is Thunderbolt working? (It’s rhetorical) So, it is quite possible to get a bigger window as long as OS allows to rebuild the resource tree. Unfortunately I don’t believe v4.19 is capable of that, it’s quite old. My suggestion to OP is to update kernel to let’s say v5.10.y (if you want LTS) and try again. – 0andriy Apr 24 '21 at 08:27
  • Do you have a setting in your BIOS for "Above 4G decoding"? Perhaps enabling it might help. See https://superuser.com/questions/1239231/what-is-above-4g-decoding – haggai_e Apr 25 '21 at 05:47
  • @haggai_e My BIOS doesn't expose this setting. Is this neccessary for virtualization? – Krustenkaese Apr 27 '21 at 08:29
  • @0andriy I switched to Ubuntu 20.04 LTS with kernel 5.8.0-50 and still get the same error – Krustenkaese Apr 27 '21 at 08:29
  • @Krustenkaese, I believe this setting is needed for the BIOS to allocate the SRIOV BAR. In theory you could still reallocate in the OS kernel. – haggai_e Apr 27 '21 at 10:53
  • @haggai_e I finally got a BIOS update and enabled SR-IOV and "Above 4G" support. Error stays the same. – Krustenkaese May 03 '21 at 09:43

0 Answers0