Issue with accepting connection using websocketpp with firewall enabled

Question

This issue is in the top 5 strangest things I have ever encountered. So please bear with me.

I am running arch linux (up to date as of 2017-03-09) with ufw firewall.

After I enable the firewall I can no longer connect to the websocketpp based application. And no, I have not blocked the port in question.

In fact if I disable the firewall I can still not connect until I restart the whole computer (just restarting the applications in question does not help).

And, UFW is also allowing all traffic on localhost but it does not make a difference if I connect locally or not.

I have verified the issue with the example echo server in the websocketpp repository too.

Check that my app is listen on port 9002

sudo netstat -lnp | grep 900
tcp6       0      0 :::9002                 :::*                    LISTEN      12695/./bin/echo_se

Here we can see that the echo server is binding to ipv6. But changing to ipv4 only does not help. Already tried.

Output from the echo server with debug

[2017-03-09 11:38:06] [devel] endpoint constructor
[2017-03-09 11:38:06] [devel] server constructor
[2017-03-09 11:38:06] [devel] asio::init_asio
[2017-03-09 11:38:06] [devel] set_message_handler
[2017-03-09 11:38:06] [devel] asio::listen
[2017-03-09 11:38:06] [devel] create_connection
[2017-03-09 11:38:06] [devel] asio con transport constructor
[2017-03-09 11:38:06] [devel] connection constructor
[2017-03-09 11:38:06] [devel] transport::asio::init
[2017-03-09 11:38:06] [devel] asio::async_accept

I'm using telnet to try and connect (since at this point I only care about the tcp handshake

telnet localhost 9002
Trying ::1...
(eventually timeout)

And If I close the echo server so no app is listening any more. Then I get (as expected)

telnet localhost 9002
Trying ::1...
Connection failed: Connection refused
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

So, now the OS notice that no one is listening and sends connection refused.

And the UWF rules:

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                  
22                         ALLOW       Anywhere                          
7000:8000/udp              ALLOW       Anywhere                  
7000:8000/tcp              ALLOW       Anywhere                  
9000:9500/tcp              ALLOW       Anywhere                  
9002                       ALLOW       Anywhere                  
22/tcp (v6)                ALLOW       Anywhere (v6)             
22 (v6)                    ALLOW       Anywhere (v6)                    
7000:8000/tcp (v6)         ALLOW       Anywhere (v6)             
9000:9500/tcp (v6)         ALLOW       Anywhere (v6)             
9002 (v6)                  ALLOW       Anywhere (v6)

I should also mention that other servers. Such as a minimal boost asio telnet server, a zmq based server etc that I have tried on the same port is working just fine.

So, I think it must be something with websocketpp but I don't understand what it could be. It feels impossible since the tcp handshake is performed below application level.

Any ideas are appreciated at this point.

For completeness. After disabling UFW and rebooting I get this from echo server:

2017-03-09 11:49:17] [devel] endpoint constructor
[2017-03-09 11:49:17] [devel] server constructor
[2017-03-09 11:49:17] [devel] asio::init_asio
[2017-03-09 11:49:17] [devel] set_message_handler
[2017-03-09 11:49:17] [devel] asio::listen
[2017-03-09 11:49:17] [devel] create_connection
[2017-03-09 11:49:17] [devel] asio con transport constructor
[2017-03-09 11:49:17] [devel] connection constructor
[2017-03-09 11:49:17] [devel] transport::asio::init
[2017-03-09 11:49:17] [devel] asio::async_accept
[2017-03-09 11:49:23] [devel] asio::handle_accept
[2017-03-09 11:49:23] [devel] connection start
[2017-03-09 11:49:23] [devel] asio connection init
[2017-03-09 11:49:23] [devel] asio connection handle pre_init
[2017-03-09 11:49:23] [devel] asio connection post_init
[2017-03-09 11:49:23] [devel] asio connection handle_post_init
[2017-03-09 11:49:23] [devel] connection handle_transport_init
[2017-03-09 11:49:23] [devel] connection read_handshake
[2017-03-09 11:49:23] [devel] asio async_read_at_least: 1
[2017-03-09 11:49:23] [devel] create_connection
[2017-03-09 11:49:23] [devel] asio con transport constructor
[2017-03-09 11:49:23] [devel] connection constructor
[2017-03-09 11:49:23] [devel] transport::asio::init
[2017-03-09 11:49:23] [devel] asio::async_accept
[2017-03-09 11:49:23] [devel] asio post init timer cancelled

And this in telnet:

telnet localhost 9002
Trying ::1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

So then everything works perfekt.

And "after sudo ufw enable && sudo ufw disable" it does not work any more...

score 1 · Accepted Answer · edited May 23 '17 at 11:46

Ok, after two days of debugging (one day before I wrote the question and then today) I have found it.

It was the backlog parameter in the listen() call to the OS that was 0. When UFW/iptables was loaded this resulted in a non functional server and a functional server otherwise.

Why the behaviour is different with iptables loaded or not is a mystery to me.

This post suggest that the backlog should have been 16 as a minimum even when set to 0. Perhaps (and this is me guessing) with iptables loaded the kernel no longer round up the backlog argument and 0 causes it to "not work". Which I can understand, since a queue of 0 makes no sense.

Anyway. Setting the backlog to 64 solved my problem.

websocketPpServerObject.set_listen_backlog(64);

Also see this github thread if you are interested in more details.

Issue with accepting connection using websocketpp with firewall enabled

1 Answers1