mbedtls DTLS server running on 64-bit desktop but not on embedded system

Question

I am trying to replace the UDP communication with a DTLS server on my application.

I have a DTLS server running on my computer (linux mint) using mbedtls. The application is reading/writing without crashing. (Though closing the connection is not yet properly implemented). When I run my application on the embedded system (32-bit geode) the application crashes.

A segmentation fault occurs when the read function tries to access part of the mbedtls_ssl_ctx (that has been overwritten) as you can see from the gdb output.

mbedtls_ssl_read and mbedtls_ssl_write are called from different threads and reading/writing concurrently. To make it thread safe I enabled MBEDTLS_THREADING_C and MBEDTS_THREADING_PTHREAD. (Is this correct?)

The setup of the DTLS server is implemented here. It is set up when a new "Node" (Client) connects.

Can someone please tell if there is an error with the setup of the DTLS server in general?
Or what might cause this error?

gdb output:

Thread 3 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 3421.3485]
0x08912351 in mbedtls_debug_print_ret (ssl=0xa76d5c3c, level=2, 
    file=0x8ad259c "/home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/ssl_msg.c", line=1853, text=0x8ad2bb9 "ssl->f_recv(_timeout)", ret=55)
    at /home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/debug.c:115
115         NULL == ssl->conf->f_dbg ||
(gdb) bt
#0  0x08912351 in mbedtls_debug_print_ret (ssl=0xa76d5c3c, level=2, 
    file=0x8ad259c "/home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/ssl_msg.c", line=1853, text=0x8ad2bb9 "ssl->f_recv(_timeout)", ret=55)
    at /home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/debug.c:115
#1  0x08925319 in mbedtls_ssl_fetch_input (ssl=0xa76d5c3c, nb_want=13)
    at /home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/ssl_msg.c:1853
#2  0x0892939b in ssl_get_next_record (ssl=0xa76d5c3c)
    at /home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/ssl_msg.c:4379
#3  0x08928364 in mbedtls_ssl_read_record (ssl=0xa76d5c3c, update_hs_digest=1)
    at /home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/ssl_msg.c:3727
#4  0x0892a76c in mbedtls_ssl_read (ssl=0xa76d5c3c, 
    buf=0x8d01ea0 <Security::DtlsServer::m_bfr> "", len=16383)
    at /home/github/mt_feature_security/implementation/dune/dune/vendor/libraries/mbedtls/library/ssl_msg.c:5265
#5  0x08701f5f in Security::DtlsServer::Node::read (this=0xa76d53d0)
    at /home/github/mt_feature_security/implementation/dune/dune/src/Security/DtlsServer/Node.cpp:548
#6  0x08702ec0 in Security::DtlsServer::Listener::run (this=0x8ddde40)
    at /home/github/mt_feature_security/implementation/dune/dune/src/Security/DtlsServer/Listener.cpp:94
--Type <RET> for more, q to quit, c to continue without paging--
#7  0x085d57d7 in dune_concurrency_thread_entry_point (data=0x8ddde40)
    at /home/github/mt_feature_security/implementation/dune/dune/src/DUNE/Concurrency/Thread.cpp:90
#8  0xa7fba12e in ?? () from target:/lib/libpthread.so.0
#9  0xa7ce67be in clone () from target:/lib/libc.so.6
(gdb) p *ssl
$1 = {conf = 0x66737365, state = 2037148789, renego_status = 1869772576, 
  renego_records_seen = 857761140, major_ver = 1646276913, minor_ver = 1936028793, 
  badmac_seen = 10, f_vrfy = 0x0, p_vrfy = 0x0, f_send = 0x0, f_recv = 0x0, f_recv_timeout = 0x0, 
  p_bio = 0x0, session_in = 0x0, session_out = 0x0, session = 0x0, session_negotiate = 0x0, 
  handshake = 0x0, transform_in = 0x0, transform_out = 0x0, transform = 0x0, 
  transform_negotiate = 0x0, p_timer = 0x0, f_set_timer = 0x0, f_get_timer = 0x0, in_buf = 0x0, 
  in_ctr = 0x0, in_hdr = 0x0, in_len = 0x0, in_iv = 0x0, in_msg = 0x0, in_offt = 0x0, 
  in_msgtype = 0, in_msglen = 0, in_left = 0, in_epoch = 0, next_record_offset = 0, 
  in_window_top = 0, in_window = 0, in_hslen = 0, nb_zero = 0, keep_current_message = 0, 
  disable_datagram_packing = 0 '\000', out_buf = 0x0, out_ctr = 0x0, out_hdr = 0x0, out_len = 0x0, 
  out_iv = 0x0, out_msg = 0x0, out_msgtype = 0, out_msglen = 0, out_left = 0, 
  cur_out_ctr = "\000\000\000\000\000\000\000", mtu = 0, client_auth = 0, hostname = 0x0, 
  alpn_chosen = 0x0, cli_id = 0x0, cli_id_len = 0, secure_renegotiation = 0, verify_data_len = 0, 
  own_verify_data = '\000' <repeats 11 times>, peer_verify_data = '\000' <repeats 11 times>, 
  f_export_keys = 0x0, p_export_keys = 0x0}
(gdb) 

Answer 1

According to the Mbed TLS knowledge base:

a single thread should only use or access one context at a same time, unless:

• The documentation for the functions that access the shared context explicitly states the function is thread-safe, or
• You perform explicit locking yourself (perhaps in a wrapper function).

The article goes on describing some cases that are thread-safe. SSL contexts are not one of them. You're not allowed to use the same SSL context in multiple threads, unless you do your own locking.

The typical way to distribute loads across threads is to have different threads responsible for different connections, but each connection only handled by one thread at a time. This works with Mbed TLS provided that you enable MBEDTLS_THREADING_C: it'll handle concurrency for the shared resources used during the handshake (signature keys, session tickets, session cache). You can't have one thread writing while another thread reads to the same connection.