Libevent writes to the socket only after second buffer_write

Question

Libevent is great and I love it so far. However, on a echo server, the write only sends to the socket on a second write. My writing is from another thread, a pump thread that talks to a db and does some minimal data massaging.

I verified this by setting up a callback for the write:

bufferevent_setcb( GetBufferEvent(), DataAvailable, DataWritten, HandleSocketError, this );

calling bufferevent_flush( m_bufferEvent, EV_READ|EV_WRITE, BEV_NORMAL ) doesn't seem to have any effect.

Here is the setup, just in case I blew it somewhere. I have dramatically simplified the overhead in my code base in order to obtain some help. This includes initialization of sockets, my thread init, etc. This is a multi-threaded app, so there may be some problem there. I start with this:

m_LibEventInstance = event_base_new();
evthread_use_windows_threads();
m_listener = evconnlistener_new_bind( m_LibEventInstance, 
         OnAccept, 
         this,
         LEV_OPT_CLOSE_ON_FREE | LEV_OPT_CLOSE_ON_EXEC | LEV_OPT_REUSEABLE, 
         -1,// no maximum number of backlog connections
         (struct sockaddr*)&ListenAddress, socketSize );

   if (!m_listener) {
          perror("Couldn't create listener");
          return false;
   }
   evconnlistener_set_error_cb( m_listener, OnSystemError );

AFAIK, this is copy and paste from samples so it should work. My OnAccept does the following:

void  OnAccept( evconnlistener* listenerObj, evutil_socket_t newConnectionId, sockaddr* ClientAddr, int socklen, void* context )
{
    // We got a new connection! Set up a bufferevent for it. 
    struct event_base*  base = evconnlistener_get_base( listenerObj );
    struct bufferevent* bufferEvent = bufferevent_socket_new( base, newConnectionId, BEV_OPT_CLOSE_ON_FREE );

   bufferevent_setcb( GetBufferEvent(), DataAvailable, DataWritten, 
                                   HandleSocketError, this );

  // We have to enable it before our callbacks will be called. 
  bufferevent_enable( GetBufferEvent(), EV_READ | EV_WRITE );

  DisableNagle( m_connectionId );
}

Now, I simply respond to data coming in and store it in a buffer for later processing. This is a multi-threaded application, so I will process the data later, massage it, or return a response to the client.

void     DataAvailable( struct bufferevent* bufferEventObj, void* arg )
{
const U32   MaxBufferSize = 8192;
   MyObj*   This = (MyObj*) arg;
   U8          data[ MaxBufferSize ];
   size_t      numBytesreceived;

   /* Read 8k at a time and send it to all connected clients. */
   while( 1 )
   {
      numBytesreceived = bufferevent_read( bufferEventObj, data, sizeof( data ) );
      if( numBytesreceived <= 0 ) // nothing to send
      {
         break;
      }

      if( This )
      {
         This->OnDataReceived( data, numBytesreceived );
      }
   }
}

the last thing that happens, once I look up my data, package into a buffer, and then on a threaded timeslice I do this:

bufferevent_write( m_bufferEvent, buffer, bufferOffset );

It never, ever sends the first time. To get it to send, I have to send a second buffer full of data.

This behavior is killing me and I have spent a lot of hours on it. Any ideas?

//-------------------------------------------------------

I finally gave up and used this hack instead... there just was not enough info to tell me why libevent wasn't writing to the socket. This works just fine.

int result = send( m_connectionId, (const char* )buffer, bufferOffset, 0 );

Answer 1

I met the problem, too! I spent one day on this problem. At last, I solved it.

When the thread you call event_base_dispatch, it will be asleep until any semaphore wakes it up. So, when it sleeps, you call bufferevent_write, the bufferevent's fd adds to the event list, but it won't be epoll until next time. So you must send semaphore to wake up the dispatch thread after you called bufferevent_write. The way you can use is set up an event bind pair socket and add it to event_base. Then send 1 byte anytime when you need to wake up the disptach thread.