1

Can somebody give me ideas about following situation (Linux 2.6.18-348.4.1.el5 ):

  1. At some point tcpdump shows [FIN, ACK] packet sent from server port to local client
  2. strace log shows no socket activity performed on that socket handle for that port near that moment (strace log properly shows the rest of communication for that client)
  3. Firewall and SELinux are stopped
  4. Problem is 100% reproducible with complex conditions which require the Server to perform network request for Kerberos authentication of another client connection.

What are potential causes which may lead tcpdump to show packet which missing in strace? Does it look more like server problem, TCP settings problem or some firewall service problem?

noonex
  • 248
  • 2
  • 10
  • 1
    Are you experiencing a problem, or is this hypothetical? – Halfgaar Jul 10 '14 at 09:53
  • Yes, this FIN packet aborts connection, which becomes a problem. I need to determine where to dig - application, TCP settings, or some problem ssytem service (like firewall) – noonex Jul 10 '14 at 09:59
  • Can you try connecting the server and client directly (or with one switch, or whatever)? Do FIN packages normally show up on `strace`? Could there perhaps be an 3rd (malicious) client sending spoofed FIN packages? – Halfgaar Jul 10 '14 at 10:11
  • No, FIN packets don't normally appear in an `strace`. You might see a call to `shutdown()` or `close()` on the socket, and the kernel would end up closing the connection, but you can't see individual packets in the TCP stream in `strace`. – Flup Jul 10 '14 at 10:20
  • Can you add a `tcpdump` showing the problem, and an `strace` made at the same time (both with timestamps)? – Flup Jul 10 '14 at 10:20
  • Unfortunately I cannot share the logs at the moment. I may try to obfuscate the addresses later, but this is rather hypothetical question regarding who to blame if such situation happens. strace has no close(), shutdown() etc calls to the socket descriptor, while has some other calls earlier (which have corresponding packets in trace log) – noonex Jul 10 '14 at 10:53
  • 1
    I am guessing multiple processes/threads have the file descriptor open, and one of them is calling `shutdown` on the socket. You may not be seeing this in `strace` simply because you are not looking at the right thread. With `lsof` it should be possible to find out, which processes has file descriptors for that socket. – kasperd Jul 10 '14 at 10:54
  • You gave me good idea to try auditd to see which process closes the socket handle, if any. Will report hopefully tomorrow – noonex Jul 10 '14 at 11:42

2 Answers2

0

Something triggers the closing of the connection on the server side. Most likely it is the application on the remote server that shuts down the connection. Possible reasons are idle disconnection feature on the server, software bugs etc.

If the remote server is behind NAT, it could also be caused by expiration of the NAT entry in the device doing the NAT.

You should run tcpdump on the server and see if the FIN packet is originated from that server. If it is, then the likely culprit is the application. If not, then it is the from firewall.

Tero Kilkanen
  • 36,796
  • 3
  • 41
  • 63
0

kasperd was right about strace not showing all threads of the process (somehow I was sure that attaching to process will monitor all threads). So auditd confirmed it is my application closing the socket (because some misterious "signal 33" happens and recv() doesn't handle EINTR properly. I will ask another question about this).

noonex
  • 248
  • 2
  • 10