2

I am writing a linux driver that talks to an Altera FPGA over PCIe. The FPGA supports both MSI and MSIx interrupts. In general, MSI interrupts appear to be working okay, however, MSIx interrupts appear to work on some computers and not others.

After a lot of searching and trial and error I was able to reproduce the problem of MSIx interrupts not working on a computer where they previously worked. I did this by disabling interrupt remapping in the kernel (passing intremap=off to the kernel).

Given that I am new to PCIe drivers, is this expected behavior?

In my driver, I am setting up MSI-X by calling:

if ((rc = pci_enable_msix_exact(pdev, irqs, NUM_IRQS)) < 0)
    printk(KERN_ERR "MSIX ENABLE FAILED: %d\n" rc);

...

for (i = 0; i < NUM_IRQS; ++i) {
    irqs[i].entry = i;
    rc = devm_request_irq(&pdev->dev, irqs[i].vector, irq_handler,
                          0, "my_driver", NULL);

    if (rc)
        printk(KERN_ERR "Failed to request irq: %d\n", i); 
}

This code works (no errors are returned) whether or not interrupt remapping is enabled. However, irq_handler is only called when interrupt remapping is enabled.

The MSI-X table on the hardware is different depending on the whether or not interrupt remapping is enabled.

For example when remapping is enabled the table looks like this:

   Vector Ctrl       Msg Data     Msg Upper Addr    Msg Addr
+---------------+---------------+---------------+---------------+
|  0x00000000   |  0x00000000   |  0x00000000   |  0xFEE00618   |
+---------------+---------------+---------------+---------------+
|  0x00000000   |  0x00000001   |  0x00000000   |  0xFEE00618   |
+---------------+---------------+---------------+---------------+
|  0x00000000   |  0x00000002   |  0x00000000   |  0xFEE00618   |
+---------------+---------------+---------------+---------------+

When remapping is disabled (intremap=off) it looks like this:

   Vector Ctrl       Msg Data     Msg Upper Addr    Msg Addr
+---------------+---------------+---------------+---------------+
|  0x00000000   |  0x00004153   |  0x00000000   |  0xFEE0F00C   |
+---------------+---------------+---------------+---------------+
|  0x00000000   |  0x00004163   |  0x00000000   |  0xFEE0F00C   |
+---------------+---------------+---------------+---------------+
|  0x00000000   |  0x00004173   |  0x00000000   |  0xFEE0F00C   |
+---------------+---------------+---------------+---------------+

What's going on here? Could there be something wrong with the FPGA? How can the driver know whether or not MSI-X interrupts will work if they appear to be granted without issue in both cases? Any help or questions that steer me in the right direction would be greatly appreciated.

Sam Protsenko
  • 14,045
  • 4
  • 59
  • 75
dylan3287
  • 51
  • 4
  • Please, switch to a new API, I.e. pci_alloc_irq_vectors(). – 0andriy Oct 18 '16 at 16:55
  • Unfortunately, for the time being, I am stuck with linux kernel version 4.7 which does not support that API. – dylan3287 Nov 28 '16 at 22:09
  • I see. Do you have any progress with your question? Btw, have you checked that you have no conflicts in address space for MSIx? – 0andriy Nov 28 '16 at 22:14
  • No good progress... I discovered that my code only works on computers that have VT-d enabled in the bios. As far as conflicts in address space, I'm not sure I fully understand your question. How might I check that? – dylan3287 Nov 28 '16 at 22:37
  • `cat /proc/iomem` for the first? – 0andriy Nov 28 '16 at 22:39
  • I don't think I see any overlapping memory ranges. `92300000-923fffff : PCI Bus 0000:02 92300000-92300fff : 0000:02:00.0 92300000-92300fff : fpga 92301000-923013ff : 0000:02:00.0 92301000-923013ff : fpga ` – dylan3287 Nov 28 '16 at 22:51

0 Answers0