How to avoid Undertow Connection RESET in apache benchmark test?

Question

Using apache benchmarking 100K request 20K concurrent users:

   $ ab -n 100000 -c 20000 http://localhost:8080/mrs/ping
    Completed 10000 requests
    Completed 20000 requests
    Completed 30000 requests
    Completed 40000 requests
    Completed 50000 requests
    Completed 60000 requests
    Completed 70000 requests
    Completed 80000 requests
    Completed 90000 requests
    apr_socket_recv: Connection reset by peer (104)  <<< HOW to overcome??

Below is the Undertow (version 1.2.6 + xnio-api 3.3.1) PingServer:

public class UndertowPingServer {

    private static Logger log = Logger.getLogger(UndertowPingServer.class);

    public static void main(String[] args) throws ServletException {

        PathHandler path = Handlers.path()
                .addPrefixPath("/mrs/ping", new HttpHandler() {
                    @Override
                    public void handleRequest(HttpServerExchange exchange) throws Exception {
                        exchange.getResponseHeaders().put(
                                Headers.CONTENT_TYPE, "text/plain");
                        exchange.getResponseSender().send("Server Time:" + new Date().toString() + "\n\n");
                    }
                });
Undertow.Builder builder = Undertow.builder()
   .setHandler(path)
   .addHttpListener(8080, "0.0.0.0")
   .setBufferSize(1024 * 16)
//this seems slightly faster in some configurations
  .setIoThreads(Runtime.getRuntime().availableProcessors() * 2) 
                    .setSocketOption(Options.BACKLOG, 500000)
                    .setWorkerThreads(2000)
//don't send a keep-alive header for HTTP/1.1 requests, as it is not required
                    .setServerOption(UndertowOptions.ALWAYS_SET_KEEP_ALIVE, false); 
            Undertow server = builder.build();
            server.start();
            log.info("micro-service running!");
        }
    }

All the needed linux kernel sockets and thread settings via sysctl are already done. That is why it can do the first 90K request with 20k users without issue.

Issue did NOT initially pops up in Jetty 9.x benchmarking. Therefore, I suspected Undertow at first. — Matthew Ong, May 28 '15 at 06:32

score 1 · Answer 1 · answered Aug 27 '18 at 17:09

1

You can add -r parameter to ab to prevent it exitting.

from https://httpd.apache.org/docs/2.4/programs/ab.html

-r

Don't exit on socket receive errors.

Connections will start to timeout and can prolong testing time by a timeout period.

answered Aug 27 '18 at 17:09

nio

5,141
2
24
35

score 0 · Answer 2 · answered May 28 '15 at 03:51

0

I just FOUND OUT similar issue from: blog.scene.ro/posts/apache-benchmark-apr_socket_recv

sudo sysctl -w net.ipv4.tcp_syncookies=0

This did the job. NO MORE Connection reset by peer (104). Guess this might not be a undertow or xnio-api issue.

answered May 28 '15 at 03:51

Matthew Ong

104
8

It is not a good idea to disable TCP Syn Cookies unless you're okay with getting downed by a DoS attack. – bluesmoon Feb 26 '17 at 01:17

score 0 · Answer 3 · answered May 28 '15 at 06:27

0

Another observation:

Undertow.Builder builder = Undertow.builder()
.setSocketOption(Options.BACKLOG, 100000) // << Does impact the REST

and also MAY related to:

$ sysctl -a | grep -i sync
...
net.ipv4.tcp_max_syn_backlog = 100000

answered May 28 '15 at 06:27

Matthew Ong

104
8

How to avoid Undertow Connection RESET in apache benchmark test?

Using apache benchmarking 100K request 20K concurrent users:

Below is the Undertow (version 1.2.6 + xnio-api 3.3.1) PingServer:

3 Answers3