1

I'm writing a tcp server in c that uses epoll() i/o multiplexing to manage concurrent connections. I want to timeout connections that have been idle for more than an allowed time.

So far I keep a last_active time_t variable associated with each connection, which I update to the current time in the event handler. Before doing that, I check if more than the allowed time has ellapsed since last event and if so I terminate the connection.

So far so good but it's not really what I want because the timeout is only triggered on the first out-of-time event, but if the connection remains inactive, my code doesn't detect it until it becomes "active" again.

The way I've seen this done in select() based servers is by linearly traversing the interest set during each iteration of the event loop and purge the inactive connections there. This is not a problem in select because you already have to do this traversal anyways, but I use epoll() precisely to avoid having to do this. If I do this, epoll is no better than select.

I've also looked into socket options, the closest thing I found was SO_RCVTIMEO, which makes read()/recv() return an error if it's been waiting more than the specified time. But since I'm working with i/o multiplexing and the sockets are in nonblock mode this makes no sese because the sockets don't block.

I would appreciate any insight on how to solve this. Thank you very much.

Mihai
  • 509
  • 5
  • 14
  • 1
    If you use `SO_RCVTIMEO`, do you get an `epoll` notification when a connection does time out? What options are you using for `epoll`? Are you using `EPOLLHUP` or `EPOLLRDHUP`? See https://stackoverflow.com/questions/27175281/epollrdhup-not-reliable – Andrew Henle Aug 10 '18 at 13:07
  • @AndrewHenle I've turned on `SO_RCVTIMEO` on client sockets and I've also set epoll to listen to `EPOLLHUP` and `EPOLLRDHUP`. After opening a connection via netcat and waiting idle for the specified timeout no epoll event gets triggered and the read() call doesn't return error `EAGAIN` nor `EWOULDBLOCK`, even if I send some data after the timeout. It looks like `SO_RCVTIMEO` has no effect on non-blocking sockets (?). – Mihai Aug 10 '18 at 14:20

1 Answers1

2

Since you know the last_active time for each socket, you can compute the time at which the next socket should be timed out (assuming no more I/O happens in the interim period) and pass in a timeout argument to epoll_wait() to cause it to wake up at that time so you can perform the connection-close.

That leaves the other part of the problem -- you want to be able to perform that computation without iterating across all of the sockets on every iteration of your event loop.

You can do that by maintaining a data structure (such as a priority queue) that supports finding the lowest-priority element in an efficient (e.g. O(1) or O(log(N)) manner. In this case you can use the socket's last_active value as its priority-value. Then before each iteration of your event-loop, you consult the data structure to find out which socket has the lowest-priority (aka which socket will be the next one to time out and need to be disconnected, if no further traffic occurs on it), and use that to set your epoll_wait() timeout.

Note that in order to maintain the data structure you'll need to update it every time a socket sends or receives data (to adjust its priority to reflect its fresh activity, by removing the socket from the structure and then re-inserting it with an updated/current last_time/priority value), however that is also a O(log(N)) operation so the overhead shouldn't be too high.

Jeremy Friesner
  • 70,199
  • 15
  • 131
  • 234
  • 1
    Btw if you're worried about having to remove-and-reinsert entries from the priority queue too often, you could add an optimization where you only do the remove-and-reinsert if a socket's `last_value` value has actually changed since the previous `send()` or `recv()` call -- and then trade off some amount of timing accuracy for better efficiency by storing your `last_value` values in larger-grained units (such as seconds or minutes rather than e.g. milliseconds or microseconds) so that they don't change so often. (This probably isn't really necessary, though, since O(log(N)) is still fast) – Jeremy Friesner Aug 10 '18 at 15:02