Troubleshooting nginx requests not reaching server

Question

When I browse from other servers using

wget -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" --spider http://server_ip/page --no-hsts

This is the response

Spider mode enabled. Check if remote file exists.
--2021-07-25 09:03:09--  http://server_ip/page
Connecting to 45.76.11.115:80... connected.
 HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

Nginx default conf is

server {
   listen 80 default_server;
   server_name _;
   return 404;
   location ~ /\.ht {
      deny all; 
   }
}

There are no entries in both error or access log. However it's receiving requests from some unknown servers like

some_public_ip - - [25/Jul/2021:08:53:14 +0800] "GET / HTTP/1.1" 200 399 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36" "1.68"

By checking programs listening at 80 using

lsof -nP -iTCP -sTCP:LISTEN

I get only nginx though.

nginx     19182     root    6u  IPv4 19898878      0t0  TCP *:80 (LISTEN)
nginx     19182     root    7u  IPv4 19898879      0t0  TCP *:8443 (LISTEN)
nginx     19183 www-data    6u  IPv4 19898878      0t0  TCP *:80 (LISTEN)
nginx     19183 www-data    7u  IPv4 19898879      0t0  TCP *:8443 (LISTEN)

However tcpdump showing it's receiving request

09:13:06.520109 IP local_ip.42330 > server_ip.80: Flags [.], ack 1, win 21, options [nop,nop,TS val 3610668715 ecr 3536888037], length 0
09:13:06.523627 IP local_ip.42330 > server_ip.80: Flags [P.], seq 1:173, ack 1, win 21, options [nop,nop,TS val 3610668715 ecr 3536888037], length 172: HTTP: HEAD / HTTP/1.1
09:13:06.523668 IP server_ip.80 > local_ip.42330: Flags [.], ack 173, win 508, options [nop,nop,TS val 3536888310 ecr 3610668715], length 0
09:13:06.523741 IP server_ip.80 > local_ip.42330: Flags [P.], seq 1:58, ack 173, win 508, options [nop,nop,TS val 3536888310 ecr 3610668715], length 57: HTTP
09:13:06.523781 IP server_ip.80 > local_ip.42330: Flags [F.], seq 58, ack 173, win 508, options [nop,nop,TS val 3536888310 ecr 3610668715], length 0
09:13:06.790067 IP local_ip.42330 > server_ip.80: Flags [.], ack 58, win 21, options [nop,nop,TS val 3610668990 ecr 3536888310], length 0
09:13:06.795185 IP local_ip.42330 > server_ip.80: Flags [R.], seq 173, ack 59, win 21, options [nop,nop,TS val 3610668991 ecr 3536888310], length 0
09:13:06.800086 IP local_ip.42330 > server_ip.80: Flags [R], seq 1666043585, win 0, length 0

How to fix this?

Update #1

Tried tcp dump again with

tcpdump -vvXX -n port 80

and output

    local_ip.48914 > server_ip.80: Flags [S], cksum 0xf7e9 (correct), seq 2727175320, win 42340, options [mss 1318,sackOK,TS val 3613618415 ecr 0,nop,wscale 11], length 0
        0x0000:  5600 0302 ccd0 fe00 0302 ccd0 0800 4500  V.............E.
        0x0010:  003c 5f00 0000 3806 875c 6adc f8c4 2d4c  .<_...8..\j...-L
        0x0020:  0b73 bf12 0050 a28d 6498 0000 0000 a002  .s...P..d.......
        0x0030:  a564 f7e9 0000 0204 0526 0402 080a d763  .d.......&.....c
        0x0040:  70ef 0000 0000 0103 030b                 p.........
    10:02:16.252990 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        server_ip.80 > local_ip.48914: Flags [S.], cksum 0x9c8e (incorrect -> 0xbb3e), seq 3157700385, ack 2727175321, win 65160, options [mss 1460,sackOK,TS val 3539838101 ecr 3613618415,nop,wscale 7], length 0
        0x0000:  fe00 0302 ccd0 5600 0302 ccd0 0800 4500  ......V.......E.
        0x0010:  003c 0000 4000 4006 9e5c 2d4c 0b73 6adc  .<..@.@..\-L.sj.
        0x0020:  f8c4 0050 bf12 bc36 af21 a28d 6499 a012  ...P...6.!..d...
        0x0030:  fe88 9c8e 0000 0204 05b4 0402 080a d2fd  ................
        0x0040:  a495 d763 70ef 0103 0307                 ...cp.....
    10:02:16.615737 IP (tos 0x0, ttl 56, id 24419, offset 0, flags [none], proto TCP (6), length 52)
        local_ip.48914 > server_ip.80: Flags [.], cksum 0xe6f3 (correct), seq 1, ack 1, win 21, options [nop,nop,TS val 3613618810 ecr 3539838101], length 0
        0x0000:  5600 0302 ccd0 fe00 0302 ccd0 0800 4500  V.............E.
        0x0010:  0034 5f63 0000 3806 8701 6adc f8c4 2d4c  .4_c..8...j...-L
        0x0020:  0b73 bf12 0050 a28d 6499 bc36 af22 8010  .s...P..d..6."..
        0x0030:  0015 e6f3 0000 0101 080a d763 727a d2fd  ...........crz..
        0x0040:  a495                                     ..
    10:02:16.629335 IP (tos 0x0, ttl 56, id 24423, offset 0, flags [none], proto TCP (6), length 227)
        local_ip.48914 > server_ip.80: Flags [P.], cksum 0x36e5 (correct), seq 1:176, ack 1, win 21, options [nop,nop,TS val 3613618810 ecr 3539838101], length 175: HTTP, length: 175
        HEAD /url HTTP/1.1
        User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
        Accept: */*
        Accept-Encoding: identity
        Host: server_ip
        Connection: Keep-Alive
        
        0x0000:  5600 0302 ccd0 fe00 0302 ccd0 0800 4500  V.............E.
        0x0010:  00e3 5f67 0000 3806 864e 6adc f8c4 2d4c  .._g..8..Nj...-L
        0x0020:  0b73 bf12 0050 a28d 6499 bc36 af22 8018  .s...P..d..6."..
        0x0030:  0015 36e5 0000 0101 080a d763 727a d2fd  ..6........crz..
        0x0040:  a495 4845 4144 202f 7572 6c20 4854 5450  ..HEAD./url.HTTP
        0x0050:  2f31 2e31 0d0a 5573 6572 2d41 6765 6e74  /1.1..User-Agent
        0x0060:  3a20 4d6f 7a69 6c6c 612f 342e 3020 2863  :.Mozilla/4.0.(c
        0x0070:  6f6d 7061 7469 626c 653b 204d 5349 4520  ompatible;.MSIE.
        0x0080:  362e 303b 2057 696e 646f 7773 204e 5420  6.0;.Windows.NT.
        0x0090:  352e 313b 2053 5631 290d 0a41 6363 6570  5.1;.SV1)..Accep
        0x00a0:  743a 202a 2f2a 0d0a 4163 6365 7074 2d45  t:.*/*..Accept-E
        0x00b0:  6e63 6f64 696e 673a 2069 6465 6e74 6974  ncoding:.identit
        0x00c0:  790d 0a48 6f73 743a 2034 352e 3736 2e31  y..Host:.server_ip
        0x00d0:  312e 3131 350d 0a43 6f6e 6e65 6374 696f  Connectio
        0x00e0:  6e3a 204b 6565 702d 416c 6976 650d 0a0d  n:.Keep-Alive...
        0x00f0:  0a                                       .
    10:02:16.629408 IP (tos 0x0, ttl 64, id 24117, offset 0, flags [DF], proto TCP (6), length 52)
        server_ip.80 > local_ip.48914: Flags [.], cksum 0x9c86 (incorrect -> 0xe2e4), seq 1, ack 176, win 508, options [nop,nop,TS val 3539838478 ecr 3613618810], length 0
        0x0000:  fe00 0302 ccd0 5600 0302 ccd0 0800 4500  ......V.......E.
        0x0010:  0034 5e35 4000 4006 402f 2d4c 0b73 6adc  .4^5@.@.@/-L.sj.
        0x0020:  f8c4 0050 bf12 bc36 af22 a28d 6548 8010  ...P...6."..eH..
        0x0030:  01fc 9c86 0000 0101 080a d2fd a60e d763  ...............c
        0x0040:  727a                                     rz
    10:02:16.629513 IP (tos 0x0, ttl 64, id 24118, offset 0, flags [DF], proto TCP (6), length 109)
        server_ip.80 > local_ip.48914: Flags [P.], cksum 0x9cbf (incorrect -> 0xb393), seq 1:58, ack 176, win 508, options [nop,nop,TS val 3539838478 ecr 3613618810], length 57: HTTP
        0x0000:  fe00 0302 ccd0 5600 0302 ccd0 0800 4500  ......V.......E.
        0x0010:  006d 5e36 4000 4006 3ff5 2d4c 0b73 6adc  .m^6@.@.?.-L.sj.
        0x0020:  f8c4 0050 bf12 bc36 af22 a28d 6548 8018  ...P...6."..eH..
        0x0030:  01fc 9cbf 0000 0101 080a d2fd a60e d763  ...............c
        0x0040:  727a 0000 1204 0000 0000 0000 0300 0000  rz..............
        0x0050:  8000 0400 0100 0000 0500 ffff ff00 0004  ................
        0x0060:  0800 0000 0000 7fff 0000 0000 0807 0000  ................
        0x0070:  0000 0000 0000 0000 0000 01              ...........
    10:02:16.629551 IP (tos 0x0, ttl 64, id 24119, offset 0, flags [DF], proto TCP (6), length 52)
        server_ip.80 > local_ip.48914: Flags [F.], cksum 0x9c86 (incorrect -> 0xe2aa), seq 58, ack 176, win 508, options [nop,nop,TS val 3539838478 ecr 3613618810], length 0
        0x0000:  fe00 0302 ccd0 5600 0302 ccd0 0800 4500  ......V.......E.
        0x0010:  0034 5e37 4000 4006 402d 2d4c 0b73 6adc  .4^7@.@.@--L.sj.
        0x0020:  f8c4 0050 bf12 bc36 af5b a28d 6548 8011  ...P...6.[..eH..
        0x0030:  01fc 9c86 0000 0101 080a d2fd a60e d763  ...............c
        0x0040:  727a                                     rz
    10:02:16.913890 IP (tos 0x0, ttl 56, id 24502, offset 0, flags [none], proto TCP (6), length 52)
        local_ip.48914 > server_ip.80: Flags [.], cksum 0xe361 (correct), seq 176, ack 58, win 21, options [nop,nop,TS val 3613619115 ecr 3539838478], length 0
        0x0000:  5600 0302 ccd0 fe00 0302 ccd0 0800 4500  V.............E.
        0x0010:  0034 5fb6 0000 3806 86ae 6adc f8c4 2d4c  .4_...8...j...-L
        0x0020:  0b73 bf12 0050 a28d 6548 bc36 af5b 8010  .s...P..eH.6.[..
        0x0030:  0015 e361 0000 0101 080a d763 73ab d2fd  ...a.......cs...
        0x0040:  a60e                                     ..
    10:02:16.913950 IP (tos 0x0, ttl 56, id 24506, offset 0, flags [none], proto TCP (6), length 52)
        local_ip.48914 > server_ip.80: Flags [.], cksum 0xe360 (correct), seq 176, ack 59, win 21, options [nop,nop,TS val 3613619115 ecr 3539838478], length 0
        0x0000:  5600 0302 ccd0 fe00 0302 ccd0 0800 4500  V.............E.
        0x0010:  0034 5fba 0000 3806 86aa 6adc f8c4 2d4c  .4_...8...j...-L
        0x0020:  0b73 bf12 0050 a28d 6548 bc36 af5c 8010  .s...P..eH.6.\..
        0x0030:  0015 e360 0000 0101 080a d763 73ab d2fd  ...`.......cs...
        0x0040:  a60e                                     ..
    10:02:16.913983 IP (tos 0x0, ttl 56, id 24508, offset 0, flags [none], proto TCP (6), length 52)
        local_ip.48914 > server_ip.80: Flags [R.], cksum 0xe35b (correct), seq 176, ack 59, win 21, options [nop,nop,TS val 3613619116 ecr 3539838478], length 0
        0x0000:  5600 0302 ccd0 fe00 0302 ccd0 0800 4500  V.............E.
        0x0010:  0034 5fbc 0000 3806 86a8 6adc f8c4 2d4c  .4_...8...j...-L
        0x0020:  0b73 bf12 0050 a28d 6548 bc36 af5c 8014  .s...P..eH.6.\..
        0x0030:  0015 e35b 0000 0101 080a d763 73ac d2fd  ...[.......cs...
        0x0040:  a60e

Could incorrect cksum relavant to the issue? Is there a way to check which application or process responded that request?

Ajay Singh · Answer 1 · 2021-07-25T02:49:15.257

0

Found the reason for this error, which was caused by another config file for different server_name, having

listen 80 http2;

Removing http2 from the config fixed the issue. Unfortunate that Nginx config test doesn't report this as invalid.

Following resources helped me troubleshoot;

https://stackoverflow.com/questions/16021481/nginx-not-listening-to-port-80

HTTPS works, HTTP does not

edited Jul 25 '21 at 02:49

answered Jul 25 '21 at 02:41

Ajay Singh

297
1
3
13

because http2 on port 80 is not a invalid option – djdomi Jul 25 '21 at 16:16
@djdomi Yeah, you're right. It would be invalid without no try_files? Either way, they could still test and show some error, rather than responding the request without logging. – Ajay Singh Jul 25 '21 at 16:21

Troubleshooting nginx requests not reaching server

1 Answers1