Using thread notification with Berkeley sockets

Question

I am utilizing the Berkeley sockets select function in the following way.

/*Windows and linux typedefs/aliases/includes are made here with wsa 
junk already taken care of.*/

/**Check if a socket can receive data without waiting.
\param socket The os level socket to check.
\param to The timeout value. A nullptr value will block forever, and zero
for each member of the value will cause it to return immediately.
\return True if recv can be called on the socket without blocking.*/
bool CanReceive(OSSocket& socket,
    const timeval * to)
{
    fd_set set = {};
    FD_SET(socket, &set);
    timeval* toCopy = nullptr;
    if (to)
    {
        toCopy = new timeval;
        *toCopy = *to;
    }

    int error = select((int)socket, &set, 0, 0, toCopy);
    delete toCopy;
    if (error == -1)
        throw Err(); //will auto set from errno.
    else if (error == 0)
        return false;
    else
        return true;
}

I have written a class that will watch a container of sockets (wrapped up in aother class) and add an ID to a separate container that stores info on what sockets are ready to be accessed. The map is an unordered_map.

while(m_running)
{
     for(auto& e : m_idMap)
     {
          auto id = e.first;
          auto socket = e.second;
          timeval timeout = ZeroTime; /*0sec, 0micro*/
          if(CanReceive(socket,&timeout) && 
               std::count(m_readyList.begin(),m_readyList.end(),socket) == 0)
          {
               /*only add sockets that are not on the list already.*/
               m_readyList.push_back(id);
          }
     }
}

As I'm sure many have noticed, this code run insanely fast and gobbles up CPU like there is no tomorrow (40% CPU usage with only one socket in the map). My first solution was to have a smart waiting function that keeps the iterations per second to a set value. That seemed to be fine with some people. My question is this: How can I be notified when sockets are ready without using this method? Even if it might require a bunch of macro junk to keep it portable that's fine. I can only think there might be some way to have the operating system watch it for me and get some sort of notification or event when the socket is ready. Just to be clear, I have chosen not use dot net.

The loop runs in its own thread, sends notifications to other parts of the software when sockets are ready. The entire thing is multi threaded and every part of it (except this part) uses an event based notification system that eliminates the busy waiting problem. I understand that things become OS-dependent and limited in this area.

Edit: The sockets are run in BLOCKING mode (but select has no timeout, and therefor will not block), but they are operated on in a dedicated thread. Edit: The system performs great with the smart sleeping functions on it, but not as good as it could with some notification system in place (likely from the OS).

Answer 1

First, you must set the socket non-blocking if you don't want the sockets to block. The select function does not provide a guarantee that a subsequent operation will not block. It's just a status reporting function that tells you about the past and the present.

Second, the best way to do this varies from platform to platform. If you don't want to write lots of platform specific code, you really should use a library like Boost ASIO or libevent.

Third, you can call select on all the sockets at the same time with a timeout. The function will return immediately if any of the sockets are (or were) readable and, if not, will wait up to the timeout. When select returns, it will report whether it timed out or, if not, which sockets were readable.

This will still perform very poorly because of the large number of wait lists the process has to be put on just to be immediately removed from all of them as soon as a single socket is readable. But it's the best you can do with reasonable portability.

Answer 2

How can I be notified when sockets are ready without using this method?

That's what select() is for. The idea is that your call to select() should block until at least one of the sockets you passed in to it (via FD_SET()) is ready-for-read. After select() returns, you can find out which socket(s) are now ready-for-read (by calling FD_ISSET()) and call recv() on those sockets to get some data from them and handle it. After that you loop again, go back to sleep inside select() again, and repeat ad infinitum. In this way you handle all of your tasks as quickly as possible, while using the minimum amount of CPU cycles.

The entire thing is multi threaded and every part of it (except this part) uses an event based notification system that eliminates the busy waiting problem.

Note that if your thread is blocked inside of select() and you want it to wake up and do something right away (i.e. without relying on a timeout, which would be slow and inefficient), then you'll need some way to cause select() in that thread to return immediately. In my experience the most reliable way to do that is to create a pipe() or socketpair() and have the thread include one end of the file-descriptor-pair in its ready-for-read fd_set. Then when another thread wants to wake that thread up, it can do so simply by sending a byte on the the other end of the pair. That will cause select() to return, the thread can then read the single byte (and throw it away), and then do whatever it is supposed to do after waking up.