10

Can anyone please help me to answer the questions about epoll_wait.

  1. Is it overkill to use many threads that call epoll_wait on the same fds set to serve at about 100K active sockets? or will it just be enough to create only 1 thread to perform epoll_wait?

  2. How many threads will wake up from epoll_wait when for example only one socket is ready to read data? i mean, can there be situation when 2 or more threads will wake up from epoll_wait but will have same fds in resulted events?

  3. What is the best way to organize threads in server that works with many active clients (e.g. 50K+). The best way i think is: 1 I/O Worker Thread which perfroms epoll_wait and i/o operations. + Many Data processing threads which will process the data received from I/O worker thread (can take a long time, such as any game logic) and compose new data for I/O worker thread to send to client. Am I right in this approach, or can anyone help me to find out the best way to organize this?

Thanks in advance, Valentin

Valentin
  • 860
  • 2
  • 10
  • 21

3 Answers3

12

  1. When using epoll, you want to size your thread total to the number of physical CPU cores (or hyperthread dispatch units) which you want to use for processing. Using only one thread for work means that at most one core will be active at a time.

  2. It depends on the mode of the epoll file descriptor. Events can be "edge triggered", meaning that they only happen once atomically, or "level triggered" meaning that any caller gets an event if there is space in the buffer.

  3. Not enough information to say. I'd suggest not having special purpose threads at all, for simplicity, and simply handling each event's "command" in the thread in which it is received. But obviously that depends on the nature of your application.

Andy Ross
  • 11,699
  • 1
  • 34
  • 31
  • So, if I understood correctly the best schema looks the next way: Create the number of I/O threads equal to number of cores in system and use ET epoll_wait. Each thread will have its own subset of fd's. For example 4 threads for IC2Q processor. Each thread is handling 25K connections and in total 100K. And the next question: Do I need to have separate thread that will epoll_wait listener socket and manage in what subset of id's newly accepted socket will be added? And is it thread-safe to add newly accepted fd using epoll_ctl in one thread, while other is making epoll_wait on this subset? – Valentin Jan 19 '10 at 09:19
  • 2
    I would wait on all descriptors in all threads, actually. Unless you know that you can win on cache effects by isolating related work on specific CPUs, it's usually a loss to do that kind of partitioning. You end up starving one CPU while another still has work that could be done. And yes: epoll operations are atomic (though obviously you'll need to lock any of your own bookeeping yourself). – Andy Ross Jan 19 '10 at 18:50
  • How will ET epoll_wait behave? Will all threads wake up from epoll_wait? Or just only 1 thread? If I understood correctly ET epoll_wait is atomic and will happen only once for ready fd's. For example: I have 2 fds and 2 threads waiting on epoll_wait. 1 fd becomes ready and only 1 thread will be resumed and if other fd will become ready during 1st thread deals with first fd, second thread will be resumed. Andy, Is this correct? – Valentin Jan 21 '10 at 09:27
  • @Valentin (it's very late, but) yes, I believe this is correct. – Armin Rigo May 24 '13 at 06:44
  • 2
    @Valentin Use EPOLLONESHOT if you want to guarantee waking up only once per file descriptor. EPOLLET kinda achieves the same thing, but without so strong a guarantee. [This](https://raw.github.com/dankamongmen/libtorque/master/doc/mteventqueues) explains in more detail. – Craig M. Brandenburg Aug 26 '13 at 23:32
  • This answer is plain wrong. See my answer for details. – thodg Jun 21 '17 at 14:56
5

I recommend you read this article from 2006: http://www.kegel.com/c10k.html

WillOw
  • 185
  • 13
Bandi-T
  • 51
  • 1
-4

Actually this is a wrong use case of epoll.

You must absolutely not share the epoll fd between threads. Otherwise you have the possibility that one thread read part of incoming data on one fd and another thread too on the same fd without any way to know which part of the data was before the other.

Just call epoll_create in each and every thread that calls epoll_wait. Otherwise the I/O is broken.

thodg
  • 1,773
  • 15
  • 24
  • 3
    Seems to me like you're confusing the epoll fd with the fd of the actual socket. Using epoll across multiple threads is absolutely possible, in fact, that's one of the points of using it. In order not to have a race condition between the threads, you either need to use `EPOLLONESHOT` or, in newer kernels, `EPOLLEXCLUSIVE`. [This great blog](https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/) explains the details. – kralyk Sep 30 '17 at 17:59