No response from a remote linux server on specified ports

Question

I have a remote CentOS 6.5 server that is running a tomcat 8 web server (AKA servlet) on port 54321 and I can't access it from my computer, although I'm establishing a successful SSH connection (that's how all the following log/command samples occurs) and a public HTTP request on exiting Apache Web Sever on port 80.

Tomcat was successfully deployed, and here is some of the log output:

07-Apr-2015 21:21:10.286 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-54321"]
07-Apr-2015 21:21:10.301 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
07-Apr-2015 21:21:10.304 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 538 ms
07-Apr-2015 21:21:10.325 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
07-Apr-2015 21:21:10.325 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.21
07-Apr-2015 21:21:10.335 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /home/mark/apache-tomcat-8.0.21/webapps/examples
07-Apr-2015 21:21:10.901 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /home/mark/apache-tomcat-8.0.21/webapps/examples has finished in 566 ms
07-Apr-2015 21:21:10.901 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /home/mark/apache-tomcat-8.0.21/webapps/host-manager
07-Apr-2015 21:21:10.923 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /home/mark/apache-tomcat-8.0.21/webapps/host-manager has finished in 22 ms
07-Apr-2015 21:21:10.923 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /home/mark/apache-tomcat-8.0.21/webapps/docs
07-Apr-2015 21:21:10.937 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /home/mark/apache-tomcat-8.0.21/webapps/docs has finished in 14 ms
07-Apr-2015 21:21:10.937 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /home/mark/apache-tomcat-8.0.21/webapps/ROOT
07-Apr-2015 21:21:10.951 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /home/mark/apache-tomcat-8.0.21/webapps/ROOT has finished in 14 ms
07-Apr-2015 21:21:10.951 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /home/mark/apache-tomcat-8.0.21/webapps/manager
07-Apr-2015 21:21:10.970 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /home/mark/apache-tomcat-8.0.21/webapps/manager has finished in 19 ms
07-Apr-2015 21:21:10.973 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-54321"]
07-Apr-2015 21:21:10.978 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 674 ms

When I run lsof -i, I can see it's listening:

java      32818    mark   49u  IPv4 78494638      0t0  TCP *:54321 (LISTEN)
java      32818    mark   68u  IPv4 78494652      0t0  TCP localhost:mxi (LISTEN)

But when I run nmap localhost, the port does not appear:

Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 993 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
111/tcp  open  rpcbind
199/tcp  open  smux
631/tcp  open  ipp
1311/tcp open  rxmon

I've checked the iptables, it's wide open:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

And here is the system-config-firewall:

# Configuration file for system-config-firewall

--disabled
--service=ssh

I've also tried nc -l 54321, and I've got the same issue (nc is listening, but can't be found via nmap).

So when I try connecting remotely to tomcat via my browser using 212.***.***.***:54321, I get "the server is not responding" (ERR_CONNECTION_TIMED_OUT).

Where's the problem?

EDIT:
After I ran nmap -p 54321 localhost I saw that the port is actually opened. So why I cant access it?

EDIT 2:
When I use remote nmap test (not from a localhost), I am filtered somehow:

PORT     STATE    SERVICE
22/tcp   open     ssh
25/tcp   filtered smtp
80/tcp   open     http
111/tcp  filtered rpcbind
199/tcp  filtered smux
631/tcp  filtered ipp
1311/tcp filtered rxmon

Same goes for tomcat on port -54321

If you run `nmap` with default options it will probably not scan this particular port. Try to test it with `nmap -p 54321` — baf, Apr 07 '15 at 19:42
@baf You are right, I followed your instruction and the port is open and named "unknown" service. — fatsokol, Apr 08 '15 at 05:04
@fatsokol are your computer and "server" connected to the same network or are they crossing any network gear like routers/firewalls? — Droopy4096, Apr 08 '15 at 05:08
my computer and the server both on a different network each (we are connecting via WWW) — fatsokol, Apr 08 '15 at 14:40
If you stop tomcat and use `nc -l -p 54321` and can't connect with straight telnet then based on the info given you have something external interfering. If `54321` is truly the port you are using I wonder if a firewall on a company network or an ISP is blocking that port since it is a well known BackOrifice port. — Michael Petch, Apr 08 '15 at 17:11

shodanshok · Accepted Answer · 2015-04-08T10:44:14.810

0

Maybe SELINUX is blocking the Tomcat process to start listening on port 54321. To check if this is the case, issue the command setenforce 0, restart tomcat, and re-try to connect. For more information: http://wiki.centos.org/HowTos/SELinux

If it not SELINUX related, try to connect to localhost issuing the command telnet 127.0.0.1 54321. If the telnet opens, then something is dropping your packet when coming from external sources. As iptables is completely disabled, the more probable thing is that an external firewall (eg: your company's gateway firewall and/or the hosting company one) is blocking traffic to port 54321.

To be sure about that, issue the command tcpdump -i any -nnn port 54321 on your server and try to connect from an external origin. If tcpdump does not log anything, something between you and your server (read: a gateway firewall) is dropping you packets.

edited Apr 08 '15 at 10:44

answered Apr 07 '15 at 19:35

shodanshok

47,711
7
111
180

it's quite clear that 54321 port bind happens successfully from the lsof output. However to prove anything one must check /var/log/audit/audit.log for any entries related to this. Or just run blunt "audit2allow < /var/log/audit/audit.log" and see whether resulting policy includes any hints about missing policy for netowrk connections of this sort. – Droopy4096 Apr 08 '15 at 05:05
after I ran `nmap -p 54321 localhost` I saw that the port is actually opened. So why I cant access it? – fatsokol Apr 08 '15 at 15:00
As stated above, something between you and your server is blocking the connection. Is not possible for us to identify precisely what is doing that, but the first things to check are your gateway firewall and your ISP/provider one. – shodanshok Apr 08 '15 at 17:10
After all, there was a filtering somewhere in my organization that I didn't know exists. This is one of the drawbacks when you working too much on your own, being misanthrope and trying to fix everything yourself. – fatsokol Apr 19 '15 at 11:02

No response from a remote linux server on specified ports

1 Answers1