I wrote a simple udp_recv test demo, and I found one strange thing
If I set the CPU affinity to cpu0, the cpu usage is 5%, but after setting affinity to other cpu, the cpu usage increased to 9%~12%,especially set to cpu20, the cpu usage is more than 25%!
I use perf -e cpu-lock top -p xxxx to check the hotpot functions.
When I set affinity to cpu0, the cpu usage is 5%, and I got this:
Samples: 20K of event 'cpu-clock', Event count (approx.): 742033834
20.41% [kernel] [k] fget_light
11.84% [kernel] [k] copy_user_enhanced_fast_string
8.63% [kernel] [k] sock_recvmsg
6.49% [kernel] [k] _raw_spin_unlock_irqrestore
5.81% [kernel] [k] aa_revalidate_sk
3.93% [kernel] [k] SYSC_recvfrom
3.85% [kernel] [k] sockfd_lookup_light
3.47% [kernel] [k] udp_recvmsg
Then I set CPU affinity to cpu5 , the cpu usage is 10%, and I got this:
Samples: 1K of event 'cpu-clock', Event count (approx.): 286637653
13.33% [kernel] [k] copy_user_enhanced_fast_string
10.51% [kernel] [k] _raw_spin_unlock_irqrestore
7.02% [kernel] [k] fget_light
5.55% [kernel] [k] system_call_after_swapgs
4.68% [kernel] [k] sock_recvmsg
4.65% udp_recv [.] process_udp_connection(int)
4.21% [kernel] [k] __slab_free
4.15% libc-2.15.so [.] __GI___libc_recvfrom
4.15% [kernel] [k] skb_copy_datagram_iovec
3.76% [kernel] [k] inet_recvmsg
3.45% [kernel] [k] __ticket_spin_lock
3.19% [kernel] [k] udp_recvmsg
2.91% [kernel] [k] SYSC_recvfrom
2.72% [kernel] [k] move_addr_to_user
2.38% [kernel] [k] put_compound_page.part.18
Then I set CPU Affinity to cpu20, the cpu usage is 27%, And I got this:
Samples: 238K of event 'cpu-clock', Event count (approx.): 2529242834
16.81% [kernel] [k] copy_user_enhanced_fast_string
8.46% [kernel] [k] udp_recvmsg
6.48% [kernel] [k] _raw_spin_unlock_irqrestore
4.59% [kernel] [k] skb_release_data.part.49
4.28% [kernel] [k] sock_recvmsg
4.12% [kernel] [k] __slab_free
3.28% libc-2.15.so [.] __GI___libc_recvfrom
3.14% [kernel] [k] SYSC_recvfrom
2.88% [kernel] [k] kmem_cache_free
2.85% [kernel] [k] __skb_recv_datagram
2.71% [kernel] [k] inet_recvmsg
2.71% [kernel] [k] system_call_after_swapgs
2.69% udp_recv [.] process_udp_connection(int)
2.58% [kernel] [k] fget_light
2.03% [kernel] [k] skb_free_datagram_locked
1.92% [kernel] [k] skb_release_head_state
I observed that function copy_user_enhanced_fast_string CPU percentage increased. I think this is the reason.
So why the CPU affinity impact so much?
