Accessing Hardware Performance Counters in RISC-V

Question

I would like to instrument a program to access hardware performance counters.

I have compiled a basic Rocketchip (freechips.rocketchip.system-DefaultConfig) and am using riscv-pk to run a binary. I am running a simulation of the core in Verilator, having compiled it all with largely defaults in the UCB Chipyard project.

The C for the binary is as follows:

#include <stdio.h>

#define CACHE_MISS      0x100

int loop(int n, int a) {
        int b = 0;
        for(int i=0; i<n; i++) {
                b = a + b;
        }
        return b;
}

int main() {
        int a = 1;
        int n = 200;
        int count = 0;

        printf("Configuring event monitor ...\n");
        /*
        // initialize counters here
        // should start tracking cache misses with 0x100
        __asm__ volatile("csrw mhpmevent3, %0"
                                :
                                : "r"(CACHE_MISS)
                        );
        */
        printf("Executing loop ...\n");
        loop(n, a);

        printf("Reading event counter ...\n");
        // read counters here
        __asm__ volatile("csrr %0, hpmcounter3"
                                : "=r"(count)
                        );
        printf("Cache misses: %d\n", count);
        return 0;
}

It's a basic code just to see if I can count cache misses. This code gives me an "illegal instruction" error.

z  0000000000000000 ra 00000000000101e0 sp 000000000fee9b00 gp 000000000001e560
tp 0000000000000000 t0 0000000000000000 t1 000000000000000f t2 0000000000000000
s0 000000000fee9b20 s1 0000000000000000 a0 00000000000000c8 a1 0000000000000001
a2 0000000000000000 a3 0000000000000010 a4 00000000000000c8 a5 00000000000000c8
a6 000000000000001f a7 0000000000000000 s2 0000000000000000 s3 0000000000000000
s4 0000000000000000 s5 0000000000000000 s6 0000000000000000 s7 0000000000000000
s8 0000000000000000 s9 0000000000000000 sA 0000000000000000 sB 0000000000000000
t3 0000000000000000 t4 0000000000000000 t5 0000000000000000 t6 0000000000000000
pc 00000000000101e0 va 00000000b03027f3 insn       b03027f3 sr 8000000200046020
An illegal instruction was executed!

Originally the line that read from hpmcounter3 was reading from mhpmcounter3 and resulted in the same error.

Speaking with another developer, one possible cause of this is the fact that riscv-pk runs binaries in U-mode, so the illegal instruction after a call to m* assembly instructions (which run in Machine, M-mode) is expected. So I changed the binary to comment away the first write to mphmevent3 and just did a bare read from hpmcounter3, but still get the illegal instruction error. The theory for this result was that the mcounteren and scounteren registers were not enabled, so I couldn't access the counters. However, looking inside of riscv-pk's minit.c, I find:

  // Enable user/supervisor use of perf counters
  if (supports_extension('S'))
    write_csr(scounteren, -1);
  if (supports_extension('U'))
    write_csr(mcounteren, -1);

I'm not sure, but I feel like this enables the counters.

Any thoughts on how I ought to go about accessing the hardware performance counters?

score 3 · Accepted Answer · answered Feb 08 '20 at 18:41

Yes, pk runs binaries in user mode, thus it's expected that accessing a machine-level register from user space yields a illegal instruction exception:

Attempts to access a CSR without appropriate privilege level or to write a read-only register also raise illegal instruction exceptions.

(RISC-V Privileged spec 1.12-draft 2020-01-13, Section 2.1 CSR Address Mapping Conventions)

The hpmcounter* registers are accessible from user-mode (cf. Table 2.2 Currently allocated RISC-V user-level CSR addresses, RISC-V privileged spec), if and only if they are enabled in the next higher mode:

If S-mode is implemented, the same bit positions in the scounteren register analogously control access to these registers while executing in U-mode. If S-mode is permitted to access a counter register and the corresponding bit is set in scounteren, then U-mode is also permitted to access that register.

The pk snippet that enables them looks sound. The 'S' and 'U' misa bits are defined in the RISC-V privileged spec:

The “U” and “S” bits will be set if there is support for user and supervisor modes respectively.

But note that the RISC-V privileged spec also specifies (emphasis mine):

Registers mcounteren and scounteren are WARL registers that must be implemented if U-mode and S-mode are implemented. Any of the bits may contain a hardwired value of zero, indicating reads to the corresponding counter will cause an illegal instruction exception when executing in a less-privileged mode.

Perhaps your CPU doesn't implement hpmcounter3 and has it hardwired to zero?

To troubleshoot this further you can add a breakpoint inside pk, i.e. right before write_csr(scounteren, -1) and single step assembler instructions to see if the writes actually happen. If they happened you can inspect the registers to check if hpmcounter3 is hardwired to zero or not.

Accessing Hardware Performance Counters in RISC-V

1 Answers1