0

My board has a Cavium Octeon NPU, running Linux kernel 2.6.34.10 that acts as a PCIe Root Complex. It is connected to PCIe switch, as are some other peripheral devices (Endpoints), among which there is Marvell's 9143 PCI-to_SATA controller based SSD.

When PCIe is initially enumerated, PCI driver on Octeon adds up the sizes of all the prefetchable memory resources and programs the PLIMIT and PBASE registers on the upstream switch port accordingly. In my case that address range is 0x80000000 - 0xEFFFFFFF.

After that, I would expect that address range to be inaccessible to kernel memory manager allocating for DMA buffers etc. And yet, I see the kernel, at some point starts sending SCSI requests to the SSD device, where scatter-gather list elements fall within this address range. I confirmed this, by looking at PCI analyzer trace. Naturally, when SSD controller receives such an address, it tries to access it (DMA read or write), and fails, because upstream switch port refuses to forward this request upstream to Root Complex, because it is programmed to think that this address would be downstream from it. (Interestingly enough, it mostly happens when I manipulate large files, I see that kernel allocated buffer addresses grow downward, until they dip below 0xEFFFFFFF)

Hence, the question: shouldn't PCI enumeration/rescan code, tell the kernel - these are PCI devices register addresses and therefore are off-limit for DMA buffer allocation? Or is it responsibility of each individual device driver to reserve its prefetchable memory? Marvell driver I use reserves regular memory BAR, but not the prefetcheable one. Is that a problem?

Thanks in advance and apologies for lengthy description.

Robbie Wxyz
  • 7,671
  • 2
  • 32
  • 47
Alex
  • 1
  • If you haven't already, I suggest trying the linux-pci mailing list. – caf Feb 10 '14 at 03:00
  • caf, its embarrassing to ask, but how do I do that ? Is there a portal that allows you to submit qestions to various Linux mailing lists ? – Alex Feb 10 '14 at 19:00
  • @Alex No, you send a mail to the list's mail address. – CL. Feb 10 '14 at 20:09
  • Thank you, sent email to Linux-pci@vger.kernel.org – Alex Feb 10 '14 at 20:36
  • Problem solved. It was caused by the fact that PCI switch (IDT) has 32-bit config registers, while memory addresses are 64-bit. When Linux tells Marvell to DMA to/from physical memory address, whose lower 32-bit happen to fit within upstream port prefetchable window (base...limit) - switch rejects the DMA attempt as "Unsupported request", thinking that this address is downstream from it, therefore no need to forward it to root complex. And Marvell 9143 firmware doesn't handle unsupported request well. The solution - to configure ACS on PCI switch port that are connected to Marvell chip. – Alex Jul 25 '14 at 15:25

0 Answers0