4

I have a java client that invokes a thread to hit a servlet and retrieves last few lines from logs at the server, and show the retrieved log lines on the client. Every once in a while, the log thread times out. Application server is Tomcat, but the error is intermittently reproducible across both Tomcat and Websphere, with client on Windows and server on Windows. With client on Windows and server on AIX, this problem has not occurred till now. I must mention that the code was stable for quite a few iterations, and suddenly started giving these problems.

What I have tried so far

  1. The log reading client invokes the thread every 0.1 sec (used a sleep). I tried increasing the sleep time to 5 sec in the code, but it did not help.

  2. When creating URLConnection object, I set properties like connectTimeout and readTimeout. I don't think readTimeout can be a cause, because that would have thrown a Socket exception.

3 I tried working with Tomcat configuration.

Connector port="9962" protocol="HTTP/1.1" connectionTimeout="200000" redirectPort="8445" acceptCount="30"

4 . The url connection is "disconnected" after use.

5 The stack trace seems to imply that the request never reached the application server, could this be because of some OS layer limits on connection. But in that case, there would have been an entry in the Event viewer of Windows.

 java.net.ConnectException: Connection timed out: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.<init>(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)

How would you go about diagnosing this problem? The server logs don't show anything suspicious. The client and server do not have any other networking devices to the best of my knowledge so no proxy is required, and the firewall is switched off. I have not used keep alive thus far.

Abhijeet Kashnia
  • 12,290
  • 8
  • 38
  • 50
  • I think you reach some limit of connections. Try to run `netstat` and check how hany connection do you have. – user1516873 Oct 31 '12 at 12:43
  • Make sure it isn't the Internet Connection itself failing periodically. Write a program that pings 4.2.2.1 and logs the results to a file and if you get loss of packets and it corresponds to the times the software breaks. Then call the ISP to investigate. – Eric Leschinski Oct 31 '12 at 13:18
  • @Eric This is inside the office LAN. Ping was working fine all the time. I am yet to see a single packet drop on this LAN. – Abhijeet Kashnia Oct 31 '12 at 13:35
  • Check client connection timeout and client JVM GC times. If there's a GC time greater than configured connection timeout, it will throw an error of this kind. – fglez Nov 09 '12 at 13:25

1 Answers1

4

It is difficult to predict what is causing this. However, your next step should be to try running a packet sniffer on the client and / or server to see if the TCP connection requests are making it to the windows machine.

If the problem occurs both with Tomcat and Websphere, that would imply that the cause is at a lower level; i.e. in the OSes TCP/IP stacks, firewall ... or in the network. (And if the server is running in a virtual, it could be a drop-out in the virtual networking.)

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216