Crash when sending data using async_write_some

Question

My server has been running for more than 20 hours for a long time. During this period, I stress test the server uninterruptedly. Sometimes it crashes, sometimes it does not crash. I am using Asio only Asio Only Source Code to handle my network class.

void AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>& packet)
{
    sendding_.store(true);
    auto self = shared_from_this();
    socket_.async_write_some(asio::buffer(packet->peek(), packet->readableBytes()),
        [self, packet](const std::error_code& err, std::size_t writen)
        {
            self->AsyncWriteSomeCallback(err,packet,writen);
        });
}

void  AsioChannel::AsyncWriteSomeCallback(const std::error_code& error, const std::shared_ptr<Buffer>& packet, std::size_t writen)
{
    if (error)
    {
    
        return;
    }
 
    if (writen < packet->readableBytes())
    {
        packet->retrieve(writen);
        AsyncSendData(packet);
    }
    else
    {
        if(!waitwriteq_.empty())
        {
            auto next = waitwriteq_.front();
            waitwriteq_.pop();
            AsyncSendData(next);
        }
        else
        {
            sendding_.store(false);
        }
    }
}

The crash stack is as follows

#0  0x00007f172a8255f7 in raise () from /lib64/libc.so.6
#1  0x00007f172a826e28 in abort () from /lib64/libc.so.6
#2  0x00007f172a865317 in __libc_message () from /lib64/libc.so.6
#3  0x00007f172a86cfe1 in _int_free () from /lib64/libc.so.6
#4  0x00007f172b92b49d in allocate<asio::detail::thread_info_base::default_tag> (size=168, this_thread=<optimized out>) at /opt/env/include/c++/10.2.0/ext/atomicity.h:97
#5  allocate (size=168, this_thread=<optimized out>) at asio/detail/thread_info_base.hpp:73
#6  asio_handler_allocate (size=168) at asio/impl/handler_alloc_hook.ipp:34
#7  allocate<net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)> > (h=..., s=168)
    at asio/detail/handler_alloc_helpers.hpp:70
#8  allocate (n=1, this=<synthetic pointer>) at asio/detail/handler_alloc_helpers.hpp:127
#9  allocate (handler=...) at asio/detail/reactive_socket_send_op.hpp:103
#10 async_send<asio::const_buffers_1, net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, asio::execution::any_executor<asio::execution::context_as_t<asio::execution_context&>, asio::execution::detail::blocking::never_t<0>, asio::execution::prefer_only<asio::execution::detail::blocking::possibly_t<0> >, asio::execution::prefer_only<asio::execution::detail::outstanding_work::tracked_t<0> >, asio::execution::prefer_only<asio::execution::detail::outstanding_work::untracked_t<0> >, asio::execution::prefer_only<asio::execution::detail::relationship::fork_t<0> >, asio::execution::prefer_only<asio::execution::detail::relationship::continuation_t<0> > > > (io_ex=..., handler=..., flags=0,
    buffers=..., impl=..., this=0x7f17080009f8) at asio/detail/reactive_socket_service_base.hpp:281
#11 operator()<net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, asio::const_buffers_1> (flags=0, buffers=...,
    handler=<unknown type in /libgamesh.net.so, CU 0x0, DIE 0x6e7c8>, this=<optimized out>)
    at asio/basic_stream_socket.hpp:1005
#12 initiate<asio::basic_stream_socket<asio::ip::tcp>::initiate_async_send, net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, const asio::const_buffers_1&, int> (token=<unknown type in /libgamesh.net.so, CU 0x0, DIE 0x6e78e>,
    initiation=<optimized out>) at asio/async_result.hpp:151
#13 async_initiate<net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, void(std::error_code, long unsigned int), asio::basic_stream_socket<asio::ip::tcp>::initiate_async_send, const asio::const_buffers_1&, int> (token=..., initiation=<optimized out>)
    at asio/async_result.hpp:364
#14 async_write_some<asio::const_buffers_1, net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)> > (
    handler=<unknown type in /libgamesh.net.so, CU 0x0, DIE 0x6e6e1>, buffers=..., this=0x7f1708143480)
    at asio/basic_stream_socket.hpp:855
#15 net::AsioChannel::AsyncSendData (this=0x7f1708143460, packet=...) at net/asio_channel.cpp:326
#16 0x00007f172b92b964 in net::AsioChannel::SendPacket (this=<optimized out>, packet=..., size=<optimized out>)
    at net/asio_channel.cpp:284
#17 0x00007f172b93cb1e in net::AsioServerHandler::SendPacket (this=<optimized out>, pPacket=..., size=<optimized out>) at /opt/env/include/c++/10.2.0/bits/shared_ptr_base.h:1324
#18 0x00007f172b946831 in operator() (__closure=<optimized out>) at /opt/env/include/c++/10.2.0/bits/shared_ptr_base.h:1324
#19 __invoke_impl<void, net::Asio_SendPacket(uint32_t, char const*, uint32_t)::<lambda()>&> (__f=...) at /opt/env/include/c++/10.2.0/bits/invoke.h:60
#20 __invoke_r<void, net::Asio_SendPacket(uint32_t, char const*, uint32_t)::<lambda()>&> (__fn=...) at /opt/env/include/c++/10.2.0/bits/invoke.h:110
#21 std::_Function_handler<void(), net::Asio_SendPacket(uint32_t, char const*, uint32_t)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...)
    at /opt/env/include/c++/10.2.0/bits/std_function.h:291
#22 0x00007f172c42ee26 in std::function<void ()>::operator()() const (this=<optimized out>) at /opt/env/include/c++/10.2.0/bits/std_function.h:248
#23 0x00007f172b93b0b4 in asio_handler_invoke<std::function<void()> > (function=...)
    at asio/handler_invoke_hook.hpp:85
#24 invoke<std::function<void()>, std::function<void()> > (context=..., function=...)
    at asio/detail/handler_invoke_helpers.hpp:54
#25 complete<std::function<void()> > (handler=..., function=..., this=<synthetic pointer>)
    at asio/detail/handler_work.hpp:424
#26 asio::detail::completion_handler<std::function<void ()>, asio::io_context::basic_executor_type<std::allocator<void>, 0u> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (owner=0x5011470, base=0x7f15c7b2f4a0) at asio/detail/completion_handler.hpp:73
#27 0x00007f172b932d88 in complete (bytes_transferred=0, ec=..., owner=0x5011470, this=0x7f15c7b2f4a0)
    at asio/detail/scheduler_operation.hpp:39
#28 asio::detail::scheduler::do_run_one (this=this@entry=0x5011470, lock=..., this_thread=..., ec=...)
    at asio/detail/impl/scheduler.ipp:480
#29 0x00007f172b934336 in asio::detail::scheduler::run (this=0x5011470, ec=...)
    at asio/detail/impl/scheduler.ipp:203
---Type <return> to continue, or q <return> to quit---
#30 0x00007f172b9394c6 in run (this=0x50113c8) at asio/impl/io_context.ipp:62
#31 net::AsioEventLoop::Run (this=0x50113c0) at net/asio_event_loop.cpp:30
#32 0x00007f172b942ae1 in net::EventLoopThread::Run(std::function<int ()> const&, std::function<int ()> const&) (this=0x4fe9ec0, pre=..., post=...)
    at /opt/env/include/c++/10.2.0/bits/unique_ptr.h:421
#33 0x00007f172b942e6f in __invoke_impl<void, void (net::EventLoopThread::*&)(const std::function<int()>&, const std::function<int()>&), net::EventLoopThread*&, std::function<int()>&, std::function<int()>&> (__t=<optimized out>, __f=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:89
#34 __invoke<void (net::EventLoopThread::*&)(const std::function<int()>&, const std::function<int()>&), net::EventLoopThread*&, std::function<int()>&, std::function<int()>&> (
    __fn=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:95
#35 __call<void, 0, 1, 2> (__args=<optimized out>, this=<optimized out>) at /opt/env/include/c++/10.2.0/functional:416
#36 operator()<> (this=<optimized out>) at /opt/env/include/c++/10.2.0/functional:499
#37 __invoke_impl<void, std::_Bind<void (net::EventLoopThread::*(net::EventLoopThread*, std::function<int()>, std::function<int()>))(const std::function<int()>&, const std::function<int()>&)> > (__f=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:60
#38 __invoke<std::_Bind<void (net::EventLoopThread::*(net::EventLoopThread*, std::function<int()>, std::function<int()>))(const std::function<int()>&, const std::function<int()>&)> > (__fn=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:95
#39 _M_invoke<0> (this=<optimized out>) at /opt/env/include/c++/10.2.0/thread:264
#40 operator() (this=<optimized out>) at /opt/env/include/c++/10.2.0/thread:271
#41 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (net::EventLoopThread::*(net::EventLoopThread*, std::function<int ()>, std::function<int ()>))(std::function<int ()> const&, std::function<int ()> const&)> > > >::_M_run() (this=<optimized out>) at /opt/env/include/c++/10.2.0/thread:215
#42 0x00007f172aca2f70 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#43 0x00007f172c1c8dc5 in start_thread () from /lib64/libpthread.so.0
#44 0x00007f172a8e61cd in clone () from /lib64/libc.so.6
(gdb)

Why is the response causing a crash?

Not using Boost
Compiled with g++ 10.2.0 in C++17 mode
Running libstdc++ of GCC10 for C++17 feature support
Running on Red Hat Enterprise Linux Server release 7.2 (Maipo)
Everything compiled with -O2

No particular knowledge of asio, but a crash in an allocate function often means a memory error: buffer overrun, use-after-free, double free, etc. As such the bug is very likely to be in some other part of your code, and we can't help without seeing all of it. Can you create a [mcve]. Tools like valgrind and AddressSanitizer are good too. — Nate Eldredge, Dec 31 '20 at 04:29

Crash when sending data using async_write_some

0 Answers0