Failed to read pci csr via mmap using uint64_t pointer

Question

I'm trying to reading PCI CSR (Configuration Space Register) on my system via open,mmap /dev/mem.

I met some problems when using 8 byte length reading

Here is the minimal working example of my code

#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#define FATAL                                                                 \
    do {                                                                      \
        fprintf(stderr,                                                       \
                "Error at line %d, file %s (%d) [%s]\n",                      \
                __LINE__,                                                     \
                __FILE__,                                                     \
                errno,                                                        \
                strerror(errno));                                             \
        exit(1);                                                              \
    } while(0)
#define PAGE_SIZE 4096UL
#define PAGE_MASK (PAGE_SIZE - 1)

typedef struct rw_config rw_config;

struct rw_config {
    uint64_t address;
    uint64_t data;
};

static uint64_t _mmio_read_worker(uint64_t address) {
    int fd;
    void *map_base = NULL;
    void *map_address = NULL;
    uint64_t result = 0UL;

    if((fd = open("/dev/mem", O_RDONLY | O_SYNC)) < 0) FATAL;
    // PAGE_SIZE = 4096UL
    // PAGE_MASK = (PAGE_SIZE - 1) = 4095UL
    if((map_base = mmap(NULL,
                        PAGE_SIZE,
                        PROT_READ,
                        MAP_SHARED,
                        fd,
                        (address & ~PAGE_MASK)))
       == MAP_FAILED)
        FATAL;
    map_address = map_base + (address & PAGE_MASK);

    result = *(uint64_t *)map_address;
    printf("uint32_t 0x%016x, uint64_t 0x%016lx\n",
           (*(uint32_t *)map_address),
           (*(uint64_t *)map_address));

    close(fd);
    return result;
}

void rw_worker(rw_config *cfg) {
    cfg->data = _mmio_read_worker(cfg->address);
    return;
}

int main(int argc, char *argv[]) {
    rw_config *cfg = malloc(sizeof(rw_config));

    cfg->address = 0x80000000;
    cfg->data = 0x0;

    rw_worker(cfg);

    return 0;
}

Reading the address = 0x80000000 which is pci mmio base address.

The output of my code is as follows:

uint32_t 0x0000000009a28086, uint64_t 0xffffffffffffffff

And I try to using gdb to get some information.

(gdb) printf "0x%llx\n",(*(uint64_t *)map_address)
0x10000009a28086
# before assigning 'result'
(gdb) printf "0x%llx\n",result
0x0
(gdb) next
# after assigning 'result'
(gdb) printf "0x%llx\n",result
0xffffffffffffffff
(gdb) print map_address
$2 = (void *) 0x7ffff7ffb000
(gdb) x/1xg 0x7ffff7ffb000
0x7ffff7ffb000: 0x0010000009a28086

I guess I fail to casting (void*) to *(uint64_t *), but why?

The value storage in map_address is correct, am I using the wrong way to get the value?

Answer 1

After reading the replies from other members, I read some documents that may be related to this bug, and the following are some of my insights:

• I tried testing with another address which NOT in PCI CSR(Configuration Space Register), and got the correct value. So I think this bug is related to hardware registers rather than software implementation

• In EDK II UEFI Writer Guide link, using 64bits read on PCI BAR(Base Address Register, which is a part of PCI CSR) may cause an alignment fault, you should use 2x of 32bits read to achieve 64bits read. Although in the example it is not enforced that the whole CSR has this limitation, but I think there is already a good reason for this bug.

Answer 2

PCIe config space must be read using 1, 2, or 4-byte accesses. 8-byte accesses are not permitted.

This is specified in the PCIe spec, section 2.2.7.1: "For Configuration Requests, Length[9:0] must be 00 0000 01b." (Length is specified in DW.)

My experience is that 8-byte accesses always return FF in all bytes.