Recently we have seen some shared memory read/write performance issue with our application.
In our application, we copy data from shared memory /run/shm to pinned memory, so as to achieve higher GPU transfer throughput. We observed that the copy speed is so different when copying from different files (same size).
To reproduce the issue, we copied the same source file to two files (i.e. file1, file2) in shared memory. Then we confirmed the problem by counting the time of copying file to /dev/null, and the results are as below.
numactl --cpubind=0 cp /run/shm/file1 /dev/null 0.05s user 0.28s system 94% cpu 0.355 total
numactl --cpubind=1 cp /run/shm/file1 /dev/null 0.11s user 0.22s system 91% cpu 0.360 total
numactl --cpubind=0 cp /run/shm/file2 /dev/null 0.08s user 2.15s system 93% cpu 2.383 total
numactl --cpubind=1 cp /run/shm/file2 /dev/null 0.16s user 5.44s system 97% cpu 5.748 total
We have checked the meminfo, we only used 50% of memory. Nothing was swapped out (and we explicitly disabled it).
Searched around but could not identify the issue. Finally, we rebooted the system and the issue was gone.
Does anyone know what might be the root cause?
Thanks!
