epoll_wait on several Threads faster?

Question

I am thinking of programming a tcp server based on epoll. To achieve the best performance i want to implement multi core support too. But during my researches the following question came up: Is it faster to call two epoll_wait()-Calls from two different threads, each observing their own file descriptors on a dual core? Or is this as fast as calling just one single epoll_wait() which observes all file descriptors?

Since the Kernel observes the file descriptors, i think it doens't matter how much threads i use on user space calling epoll_wait()?

How often do you expect changes to occur? – Tudor Sep 18 '12 at 17:03 — Tudor, Sep 18 '12 at 17:03

score 19 · Answer 1 · answered Sep 18 '12 at 20:27

19

You can even call epoll_wait concurrently on multiple threads for the same epoll_fd as long as you use edge-triggered (EPOLLET) mode (and be careful about synchronisation). Using that approach you can get real performance benefits on multi-core machines compared to a single-threaded epoll event loop.

I have actually done performance measurements some time ago, see my blog postings for the results:

answered Sep 18 '12 at 20:27

cmeerw

7,176
33
27

1

Why only with EPOLLET mode? – piotr Jan 31 '14 at 15:16
4

Because if you work in any other mode, then the epoll will report the same file descriptors to several threads, causing spurious wake ups. – Shachar Shemesh May 23 '14 at 08:16
This link (https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/) can help with understanding the "spurious wake ups" which the link refers to as thundering herd. – Danilo Souza Morães Jul 25 '19 at 11:46

score 0 · Answer 2 · answered Sep 18 '12 at 18:02

I suspect you're correct that the performance of epoll per se is no different in the two scenarios you mention. What could make a difference, OTOH, is that by having an event loop in each thread you don't need to context switch to a separate worker thread to handle the connection. This is e.g. how nginx works, see e.g. http://www.aosabook.org/en/nginx.html .

epoll_wait on several Threads faster?

2 Answers2