0

These days, I am working on forwarding ssl payload to upstream tcp server using nginx, and the final nginx configure comes after a lot of experiment:

stream {
    map $ssl_server_name $stream_map {
        aby3_task_1 upstream_task_1;
        aby3_task_2 upstream_task_2;
    }

    upstream upstream_task_1 {
        server 127.0.0.1:1313;
    }

    upstream upstream_task_2 {
        server 127.0.0.1:1314;
    }

    server {
        listen 8185 ssl;
        ssl_certificate /home/ttt/nginx-cfg/cert/server1.crt;
        ssl_certificate_key /home/ttt/nginx-cfg/cert/server1.key;
        proxy_pass $stream_map;
        ssl_preread off;
    }

    error_log /etc/nginx/logs/error.log debug;
}

The hardest problem during experiment is that nginx don't forward ssl payload into upstream tcp server according to server name, I worked on this problem for many days but got nothing. And yesterday my workmate removed ssl_preread on by accident then solves this problem. The default value of ssl_preread is off, so we realized that set ssl_preread to off would help us to solve the problem, this is the reason why ssl_preread off appears in the final nginx configure.
Why ssl_preread on has big impact to nginx's forward process? we collect some debug log when nginx forward runs well and runs bad, see below:

# This is the log when nginx can forward ssl payload to upstream tcp server.
2023/01/28 14:25:17 [info] 2952111#2952111: *1204 client 192.168.16.184:33868 connected to 0.0.0.0:8185
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 posix_memalign: 000055B8CEC6B700:256 @16
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 tcp_nodelay
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_do_handshake: -1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_get_error: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 epoll add event: fd:4 op:1 ev:80002001
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer add: 4: 60000:8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL handshake handler: 0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_do_handshake: 1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer del: 4: 8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 ssl preread handler
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 proxy connection handler
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 malloc: 000055B8CEC92630:448
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 posix_memalign: 000055B8CEC8D1C0:256 @16
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 malloc: 000055B8CEC87C60:16384
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 post event 000055B8CECC87A0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream map started
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream script var: "aby3_task_3"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream map: "aby3_task_3" "upstream_task_3"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream script var: "upstream_task_3"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 get rr peer, try: 1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream socket 18
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 epoll add connection: fd:18 ev:80002005
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 connect to 127.0.0.1:1315, fd:18 #1205
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 proxy connect: -2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer add: 18: 60000:8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 delete posted event 000055B8CECC87A0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_read: -1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_get_error: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer del: 18: 8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream proxy connect upstream
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 tcp_nodelay

# This is the log when nginx can't forward ssl payload to upstream tcp server.
2023/01/28 14:23:21 [info] 2950336#2950336: *1192 client 192.168.16.184:60998 connected to 0.0.0.0:8185
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 posix_memalign: 000055B8CEC6A530:256 @16
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 0
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 tcp_nodelay
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_do_handshake: -1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_get_error: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 epoll add event: fd:11 op:1 ev:80002001
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 event timer add: 11: 60000:8285270365
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL handshake handler: 0
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_do_handshake: 1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL reused session
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 event timer del: 11: 8285270365
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 ssl preread handler
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 malloc: 000055B8CEC877F0:16384
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_read: -1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_get_error: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 event timer add: 11: 30000:8285240369

Compare the two log, we find that proxy connection handler occurs only in the case that nginx can forward ssl payload to upstream tcp server, so we assume that ssl_preread on will disable the forwarding, but why, can you tell me the reason?

纪老猴子
  • 33
  • 1
  • 9

1 Answers1

0

Today I notice a sentence in the introduction of nginx_stream_ssl_preread_module, the sentence is that the ngx_stream_ssl_preread_module module (1.11.5) allows extracting information from the ClientHello message without terminating SSL/TLS. I find very little detail hidden at the end of the sentence, it is without terminating SSL/TLS, then I know the answer to this problem.
SSL termination refers to the process of decrypting encrypted traffic before passing it along to a web server, and in our case，we want to decrypt encrypted traffic and pass it along to a TCP server, so what we want to Nginx do can also be called SSL termination. According to the introduction of ngx_stream_ssl_preread_module, we know that this module doesn't support our case, so we must set ssl_preread to off. Many days before, I try to use $ssl_preread_server_name in the map block and find the forward don't take effect after many experiments, today I know that the reason is the same as this question.

纪老猴子
  • 33
  • 1
  • 9