13

I am creating a multithreaded server using epoll (edge-triggered) and non-blocking sockets. Currently I'm creating an event loop on the main thread and waiting for notifications and it works correctly
I have to choose between two approaches to make it multithreaded:

  1. Create an event loop for each thread and add the server socket's file descriptor to look for notifications on each thread. (is that possible? I mean: is epoll thread-safe?)
  2. Create a single event loop and wait for notifications. Whenever a notification is received, spawn a thread to handle it.

If I use the first method, is there a chance for multiple threads to get notified with the same event? how can I handle this situation?

What could be the best approach? Thank you.

goodolddays
  • 2,595
  • 4
  • 34
  • 51
  • 2
    possible duplicate of [Is epoll thread-safe?](http://stackoverflow.com/questions/7058737/is-epoll-thread-safe) –  Jan 29 '13 at 14:11
  • 3
    Yes, `epoll` is thread-safe. Yes, you can do it. Just search around for details, SO and Google are full of this information. –  Jan 29 '13 at 14:11
  • Thank you.I would also like to know if I can add the same file descriptor to multiple epoll loops. Can I? – goodolddays Jan 29 '13 at 14:56

5 Answers5

7

I think option 1 is more popular since the primary purpose of non-blocking IO is to avoid the overhead of create & destroy threads.

take the popular web server nginx as an example, it create multiple processes (not threads) to handle incoming events on a handle, and the process the events in the subprocess. all of them share the same listening socket. it's quite similar to option 1.

solofox
  • 261
  • 2
  • 3
3

I'm also writing a server using epoll, and I have considered the same model as you attached.

It is possible to use option 1, but it may cause "thundering herd" effect, you can read the source of nginx to find the solution. As for option 2, I deem that it is better to use thread pool instead of spawning a new thread each time.

And you can also the following model：

Main thread/process: accept incoming connection with blocking IO, and send the fd to the other threads using BlockingList or to the other processes using PIPE.

Sub threads/process: create an instance of epoll respectively, and add the incoming fd to the epoll, then processing them with non-blocking IO.

CodeSun
  • 51
  • 5
1

an event loop for each thread is the most flexible with high performance You should create an epoll fd for each event loop, there is no concern about epoll thread-safe problem.

Bert Young
  • 21
  • 2
0

epoll is thread safe, a good solution is that your main process stay in accept(2), once you get the file descriptor register that one in the epoll fd for the target thread, that means that you have a epoll queue for each thread, once you create the thread you share the epoll file descriptor as parameter in the call pthread_create(3), so when a new connection arrives, you do a epoll_ctl(...EPOLL_CTL_ADD..) using the epoll fd for the target thread and the new socket created after accept(2), make sense ?

edsiper
  • 398
  • 1
  • 4
  • Why not directly add the listening socket descriptor to the `epoll` set, when new client connection request arrives, `accept` the connection, add new connected socket descriptor to the `epoll` set, in the main thread? When real data arrives, spawn a new thread to handle it. In this way, the program looks more precise. – zeekvfu Jul 03 '13 at 17:55
0

epoll is thread-safe

hope the following code can help your

https://github.com/jingchunzhang/56vfs/blob/master/network/vfs_so_r.c

user7610
  • 25,267
  • 15
  • 124
  • 150
jingchun.zhang
  • 81
  • 2