I wrote two C++ programs to communicate data with one producer and multiple consumer using one lock-free queue. The producer write one data(about 256bytes) to the queue(peroid: 100us), and the multiple consumer read all the new data as soon as possible. When the producer begins to write fixed data to the queue, it will record the current timestamp, then write the timestamp to the data. When the consumer reads the data, it will record the current timestamp, then calculate the time difference and store the value to a fixed-length array.
I use the function clock_gettime and parameter CLOCK_REALTIME to get the current timestamp.
I use grub cmdline parameter,
transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=4 nohz=on rcu_nocb_poll nohz_full=4-10 isolcpus=4-10 rcu_nocbs=4-10
I tuned my system to network-latency profile, and reset cpus to run in the performance mode(cpupower, 2.8GHz).
I move the kernel and workqueue processes out of the isolated cpus.(confirmed) I also move all the interrupts out of the isolated cpus using irqbalance command.(confirmed)
My program is compiled on debug mode.
I run my programs with taskset command, and I can confirm that they are run on the isolated cpu cores, but there are still some big jitters that high to 1ms in about 100000 write/read.
My Server is Dell R630(E5-2680v3 x2, 16GBx2 DDR4 2400MHz) with custom os performace profile. I disable hyperthreading and vt-d option, enable node interleving. The OS is CentOS-7 latest,disable all unrelated services(include irqbalance) and disable selinux.
I cannot find any valuable information in perf record data file.
Can someone give me some hints to kill these
