How to access PCI memory from Linux kernel space by memory mapping (Kernel 3.14)

Question

I'm looking for a way to access the memory space of an PCI device (explicit BAR2 and BAR3) without using DMA and IO-mapping. I have read much documentations but I never saw a flowchart or a step by step how to. So all my tries aren't successful.

These are the steps inside pci_probe I actually try:

1. data = kzalloc( sizeof(*data) , GFP_KERNEL );
2. pci_set_drvdata(pdev, data);
3. pci_enable_device(pdev);

Now is the question what is the correct address to access BAR2+offset using writeb or readb? Or is there a another function to read/write from this space?

---

PS: An similar question regarding iomap was posted here.

Answer 1

After heavy researching, I found a way and to read and write to PCI BAR2. It seems that ioremap, pci_ioremap_bar or memremap() (Kernel 4.3+) allows CPU caching the data transferred between PCI device and Kernel space memory. This causes corrupt data. But I don't know where it finally come from.

The approach to solve this issue uses ioremap_nocache from io.h. The following code shows the PCI probe function.

static int
_pci_probe ( struct pci_dev *pdev,
             const struct pci_device_id *ent )
{
  int ret = 0;
  int i;
  unsigned long *pbas2addr;
  u8  buf8;
  u8 *mem8;

  buf8  = 0xF0;
  // put mem8 to the heap and initialize them with zeros
  mem8 = kcalloc((0x020000),sizeof(u8), GFP_KERNEL);

  // enabling the device
  ret = pci_enable_device(pdev);
  if( ret )
  {
    printk(KERN_ERR "Failed to enable PCI device.\n");
    goto no_enable;
  }

  // take ownership of pci related regions
  pci_request_regions(pdev, "expdev");

  // checking if PCI-device reachable by checking that BAR0 is defined and
  // memory mapped
  if( !(pci_resource_flags(pdev,0) & IORESOURCE_MEM) )
  {
    printk(KERN_ERR "Incorrect BAR configuration.\n");
    ret = -ENODEV;
    goto bad_bar;
  }

  // remap BAR2 avoiding the use of CPU cache
  pbas2addr = ioremap_nocache(pci_resource_start(pdev,2),
                              pci_resource_len(pdev,2));

  printk(KERN_INFO "BAR2 Addr: %p\n",pbas2addr);
  printk(KERN_INFO "BAR2 len:  %x\n",(int)pci_resource_len(pdev,2));

  // write something to BAR2
  buf8  = 0xF0;
  for ( i = 0x000000; i<0x020000; i++ )
  {
    *((u8*)pbas2addr+i) = buf8; // it's important to cast the pointer
  }

  // read back
  buf8 = 0;
  for ( i = 0x000000; i<0x020000; i++ )
  {
    mem8[i] = *((u8*)pbas2addr+i);
  }

  return 0;

bad_bar:
  pci_disable_device(pdev);
no_enable:

  return ret;
}

Additionally:

1. The access of the mapped memory using iowrite doesn't work stable. Sometimes artefactual crap are found on the PCI BAR2 memory. Maybe there are hold sequences coming with this command. I don't know.
2. In my case some address ranges of the BAR2 memory needs to be write twice. I think this is a special behaviour of this device.
3. In my case not all address ranges can be reached with 32-bit access. I think this is a special behaviour of this device too.

Answer 2

A portable way is using pci_iomap() function. It's automatically determine type of BAR and works with MMIO and PIO. Instead of writeb()/readb() you should use ioread*()/iowrite*()