Libevent bufferevent socket flush

Question

I'll keep is short. How do I flush data waiting in a bufferevent output buffer directly to the socket in a blocking manner.

Upon closing my socket wrapper class after making an asynchronous write (using evbuffer_add), libevent spits out epoll errors denoting that an attempt was made to write to an invalid fd. I need to be able to flush pending libevent data to the socket, any suggestions?

Note: the specific error is Epoll MOD(4) on fd 9 failed. Old events were 6; read change was 2 (del); write change was 0 (none): Bad file descriptor.

score 2 · Answer 1 · answered Aug 17 '13 at 17:51

2

Have you tried to disable socket lingering? Disabling socket lingering causes the socket not to wait for unsent data before closing the socket.

struct linger linger;
memset(&linger, 0, sizeof(struct linger));
retVal = setsockopt(sock, SOL_SOCKET, SO_LINGER, (const void*)&linger, sizeof(struct linger));

answered Aug 17 '13 at 17:51

The problem is that libevent doesn't actually write all the data to the socket, data is left in libevents internal buffers and as of now I close the socket while data is still waiting in the buffer. I want to flush the libevent buffer to the socket and then close it. Disabling socket lingering does not help. – Dylan Aug 18 '13 at 00:57
I don't think `bufferevent_flush` works with network sockets yet, but have you tried it? – Aug 18 '13 at 12:16
If `bufferevent_flush()` don't work then I would try setting the `BEV_OPT_CLOSE_ON_FREE` flag when creating the bufferevent. Also, bufferevents are internally reference-counted, so if the bufferevent has pending deferred callbacks when you free it, it won’t be deleted until the callbacks are done. – Aug 18 '13 at 12:22
I tried it and to no avail. Setting `BEC_OPT_CLOSE_ON_FREE` also does nothing. – Dylan Aug 19 '13 at 07:17
2

it appears the error only occurs when I free the bufferevent from inside the callback function. – Dylan Aug 19 '13 at 07:20
1

don't worry - your `BEV_OPT_CLOSE_ON_FREE` suggestion worked. posting an answer in case it may help others, thanks :) – Dylan Aug 19 '13 at 07:26

score 2 · Answer 2 · answered Aug 19 '13 at 07:39

2

For any of those who might encounter the same error, I found what my problem is - thanks to Inge Henriksen.

class ASocket
{

    // ...

    ~ASocket()
    {
        if(m_handle.bev)
        {
            bufferevent_free(m_handle.bev);
        }

        if(m_handle.fd >= 0)
            ::close(m_handle.fd);
    }

    // ...
}

Upon deleting an asynchronous socket object (ASocket), the bufferevent would be freed if it exists and the socket would be deleted - libevent would continue to operate on a closed socket. Note that bufferevent_free, as stated at http://www.wangafu.net/~nickm/libevent-book/Ref6_bufferevent.html#_freeing_a_bufferevent, but not on the Doxygen documentation page, will not free the bufferevent upon calling the bufferevent_free function but rather:

The bufferevent_free() function does, however, try to free the bufferevent as soon as possible.

This was fixed like so:

class ASocket
{

    // ...

    // If bufferevent exists, it must be created with
    // the BEV_OPT_CLOSE_ON_FREE flag.
    ~ASocket()
    {
        if(m_handle.bev)
        {
            bufferevent_free(m_handle.bev);
        }
        else
        {

            if(m_handle.fd >= 0)
                ::close(m_handle.fd);
        }
    }

    // ...
}

If the socket has a bufferevent, it is freed and libevent will close the socket once it is finished.

answered Aug 19 '13 at 07:39

Dylan

572
5
10

You should put this post in your question, just add a line called "Update:" and add this post, then delete this post and accept my answer. It is not common practice on SO to create a new post with the solution when someone else has answered your question before you. – Aug 20 '13 at 10:31
@IngeHenriksen It is, actually. Self-answering is perfectly fine. +1, and I am flagging your "not an answer" flag as invalid. – tckmn Aug 20 '13 at 13:22
I sympathize with you and am grateful for the help, but it seems counter-productive to mark and answer that isn't the right answer but pointed me in the right direction as the correct answer. The answer doesn't even have mention of `BEV_OPT_CLOSE_ON_FREE`, that is in the comments. If you update your answer to include the solution, I will gladly accept it. I concur with @Doorknob. – Dylan Aug 21 '13 at 06:32
your answer was invaluable to me but "setting BEV_OPT_CLOSE_ON_FREE" isn't as much as a straightforward answer as it could be (and I reiterate, not in the answer itself but the comments) so I decided to place a specific answer with code in case someone might make the same error. – Dylan Aug 21 '13 at 06:42
Thanks for the solution, but I can't find any blocking code in bufferevent_free on flushing pending data. bufferevent_free just clear the callback, cancel ctrl data, and decref_and_unlock bufferevent. I'm calling the evbuffer_write function myself currently to try to flush data. But write() in evbuffer_write may still have remain data. I'm working on libevent-2.0.21-stable – spin6lock Mar 24 '14 at 08:40
have you tried `bufferevent_flush(bev, EV_READ|EV_WRITE, BEV_FINISHED)`? http://www.wangafu.net/~nickm/libevent-2.0/doxygen/html/bufferevent_8h.html#a03376a35668ff9b6dc905cce0d89876d – Dylan Apr 12 '14 at 05:11

score 0 · Answer 3 · answered Dec 29 '13 at 13:12

I couldn't do it inside the callback itself, but it possible to do it using another call back see https://github.com/libevent/libevent/blob/master/sample/le-proxy.c

if (partner) {
                    /* Flush all pending data */
                    readcb(bev, ctx);

                    if (evbuffer_get_length(
                                bufferevent_get_output(partner))) {
                            /* We still have to flush data from the other
                             * side, but when that's done, close the other
                             * side. */
                            bufferevent_setcb(partner,
                                NULL, close_on_finished_writecb,
                                eventcb, NULL);
                            bufferevent_disable(partner, EV_READ);
                    } else {
                            /* We have nothing left to say to the other
                             * side; close it. */
                            bufferevent_free(partner);
                    }
             }

Libevent bufferevent socket flush

3 Answers3