boost asio SSL server error: called a function you should not call

Question

I am using the example from boost with three minor differences:

I use threads to process io_service
I have limited the protocol to > TLS v1.1
There is no password callback, because the cert key has no password

The threads simply process various connections in parallel

void server::start()
{
    for (std::size_t i = 0; i < thread_pool_size_; i++) {
        threads_.push_back(std::thread([&]() {
            io_service_.run();
        }));
    }
    for (auto & t : threads_) { 
        t.join();
    }
}

The context arguments are:

ctx.set_options(boost::asio::ssl::context::default_workarounds
               |boost::asio::ssl::context::no_sslv2
               |boost::asio::ssl::context::no_sslv3
               |boost::asio::ssl::context::no_tlsv1
               |boost::asio::ssl::context::no_tlsv1_1
               |boost::asio::ssl::context::single_dh_use);

Apart from that, a connection class controls internally a ssl_socket, which follows the following callback chain:

ctor -> start -> async_handshake -> read_header -> async_read_until -> process_header ...

If I replace the SSL socket with a plaintext socket ip::tcp::socket then everything works fine. When using the SSL socket, I keep getting called a function you should not call.

GDB shows that this originates from async_handshake. After reading this SO post I managed to obtain the error code:

(20,197,66) error:140C5042:SSL routines:ssl_undefined_function:called a function you should not call.

I'm using Boost 1.58 on Ubuntu 16.04. Any help as to why this is happening, what might be causing it, or what I could possibly have done wrong?

If it matters, I am testing with curl using the -insecure flag.

EDIT

Did try without the restrictive protocol flags, and by setting a password callback - problem still persists.

@sehe it is the client. I was hoping it was ignoring the self signed certificate, but this seems to trigger regardless of protocol or certificate — Ælex, Nov 30 '17 at 17:43
Ok. That's purely client side, so it makes no difference. (Never mind about the "server" it was hidden in the title :)) — sehe, Nov 30 '17 at 17:44
My gut says you need to provide the password callback. Have you tried adding one? Might be just the case of "blank password" !=/== "no password" (Apple knows all about that now) — sehe, Nov 30 '17 at 17:45
Do you have the code in a repo/shareable? I want to have a look but I'm not feeling well enough to retrace your steps from the example app — sehe, Nov 30 '17 at 17:52
@sehe I'm honored and I would not mind to share but this is a private repo. I can copy paste the important bits to a new repo if needed, since I am not certain that I'm doing something wrong. — Ælex, Nov 30 '17 at 18:56

score 3 · Accepted Answer · answered Dec 04 '17 at 14:11

Seems that the following line triggers the problem:

context_(io_service, boost::asio::ssl::context::tlsv12_client)

I should have paid more attention to my code (copy-paste is the root of all evil): The offending line was in the context constructor:

ctx_(io_service, boost::asio::ssl::context::tlsv12_client)

I've replaced with the server version:

context_(io_service, boost::asio::ssl::context::tlsv12_server)

The full list is hidden in the implementation header of boost. Insert big facepalm here :-)

boost asio SSL server error: called a function you should not call

EDIT

1 Answers1