4

I'm trying to start glassfish on an EBS-based AMI of Ubuntu 10.04 64-bit. I have used glassfish on non-ec2 servers with no problems, but on ec2 I get this message:

$ sudo -u glassfish bin/asadmin start-domain domain1
There is a process already using the admin port 4848 -- it probably is another instance of a GlassFish server.
Command start-domain failed.

I know that ec2 has requires that firewall rules be modified using ec2-authorize to let outside traffic thru the firewall, as I had to do to make ssh work. This still doesn't explain the port error when all I'm trying to do is start glassfish so I can try 

$ wget localhost:8080
and make sure it's working.

This is very frustrating and I'd really appreciate any help.

Thanks.

FINAL UPDATE:
Sorry if you came here looking for answers. I never figured out what was causing the problem. I created another fresh instance, installed the same stuff, and Glassfish worked perfectly. Something obviously got boned during installation, but I have no idea what. I guess it will remain a mystery.


UPDATE: Here's what I get from netstat:

# netstat -nuptl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      462/sshd        
tcp6       0      0 :::22                   :::*                    LISTEN      462/sshd        
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           483/avahi-daemon: r
udp        0      0 0.0.0.0:1194            0.0.0.0:*                           589/openvpn     
udp        0      0 0.0.0.0:37940           0.0.0.0:*                           483/avahi-daemon: r
udp        0      0 0.0.0.0:68              0.0.0.0:*                           377/dhclient3



UPDATE: One more thing... I know that the "net.ipv6.bindv6only" kernel option can cause problems with java networking, so I did set this:

# sysctl -w net.ipv6.bindv6only=0



UPDATE: I also verified that it has nothing at all to do with the port number (4848). As you can see here, when I changed the admin-listener port in domain.xml to 4949, I get a similar message:

# sudo -u glassfish bin/asadmin start-domain domain1
There is a process already using the admin port 4949 -- it probably is another instance of a GlassFish server.
Command start-domain failed.



UPDATE:
Here are the contents of /etc/hosts:

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I should mention that I have another Ubuntu Lucid 10.04 64-bit slice that is NOT hosted on ec2, and set it up the exact same way with no problems whatsoever.

Also server.log doesn't offer much insight either:

# cat ./server.log 
Nov 20, 2010 8:46:49 AM com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: JVM invocation command line:
/usr/lib/jvm/java-6-sun-1.6.0.22/bin/java
-cp
/opt/glassfishv3/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
-XX:MaxPermSize=192m
-XX:NewRatio=2
-XX:+LogVMOutput
-XX:LogFile=/opt/glassfishv3/glassfish/domains/domain1/logs/jvm.log
-Xmx512m
-client
-javaagent:/opt/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=unsafe=true,noServer=true
-Dosgi.shell.telnet.maxconn=1
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Dfelix.fileinstall.dir=/opt/glassfishv3/glassfish/modules/autostart/
-Djavax.net.ssl.keyStore=/opt/glassfishv3/glassfish/domains/domain1/config/keystore.jks
-Dosgi.shell.telnet.port=6666
-Djava.security.policy=/opt/glassfishv3/glassfish/domains/domain1/config/server.policy
-Dfelix.fileinstall.poll=5000
-Dcom.sun.aas.instanceRoot=/opt/glassfishv3/glassfish/domains/domain1
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dosgi.shell.telnet.ip=127.0.0.1
-Djava.endorsed.dirs=/opt/glassfishv3/glassfish/modules/endorsed:/opt/glassfishv3/glassfish/lib/endorsed
-Dcom.sun.aas.installRoot=/opt/glassfishv3/glassfish
-Djava.ext.dirs=/usr/lib/jvm/java-6-sun-1.6.0.22/lib/ext:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/ext:/opt/glassfishv3/glassfish/domains/domain1/lib/ext
-Dfelix.fileinstall.bundles.new.start=true
-Djavax.net.ssl.trustStore=/opt/glassfishv3/glassfish/domains/domain1/config/cacerts.jks
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Djava.security.auth.login.config=/opt/glassfishv3/glassfish/domains/domain1/config/login.conf
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Dfelix.fileinstall.debug=1
-Dorg.glassfish.web.rfc2109_cookie_names_enforced=false
-Djava.library.path=/opt/glassfishv3/glassfish/lib:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.22/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-domainname
domain1
-asadmin-args
start-domain,,,domain1
-instancename
server
-verbose
false
-debug
false
-asadmin-classpath
/opt/glassfishv3/glassfish/modules/admin-cli.jar
-asadmin-classname
com.sun.enterprise.admin.cli.AsadminMain
-upgrade
false
-domaindir
/opt/glassfishv3/glassfish/domains/domain1
-read-stdin
true
orange80
  • 198
  • 2
  • 12

5 Answers5

0

You're right - firewalling has nothing to do with the open (presumed TCP?) port.

netstat is your friend. Something similar to netstat -nupt -l will give you an indication of what processes are listening on what ports.

Let us know the output of that command, and whether it helps point you in the right direction.

Zayne S Halsall
  • 1,982
  • 15
  • 19
  • Thanks. As you can see from netstat the port doesn't seem to be in use. :( – orange80 Nov 20 '10 at 08:05
  • Your issue seems related to this: http://serverfault.com/questions/90493/used-blocked-ports-in-ubuntu - where one of the answers mentions looking at the /etc/hosts file for the culprit, based on the logs. So first check/share your app logs, second check /etc/hosts for anything missing/additional. – Zayne S Halsall Nov 20 '10 at 08:25
  • I updated above. I don't see anything to indicate that /etc/hosts is the problem at all. I was able to fire up # python -m SimpleHTTPServer and read from localhost using # wget localhost:8000 with no problems as well. – orange80 Nov 20 '10 at 08:51
  • Based on some googling, I'd recommend you try adding whatever host name is returned from `hostname`, and the IP returned when doing a `dig` or `host` against that name, into your */etc/hosts* file - just as a quick test. What I don't like about the info provided thus far, is the application is clearly complaining, yet there is nothing in the logs echoing even that same issue. Are you looking at the correct logs (I know how... complex Java apps can be :) )? – Zayne S Halsall Nov 20 '10 at 09:10
  • I'm not sure where else to look honestly. I think it's weird too, but that is the full log from one invocation of asadmin start-domain. It's the _only_ log in glassfish/domains/domain1. As a test I did put the hostname and value returned by host $hostname, but still this had no effect, even after a restart. – orange80 Nov 20 '10 at 17:25
  • Two final recommendations: 1) try get **netcat** listening on the port glassfish is claiming is being used; 2) approach the glassfish developers through their forum or whatever medium they have available. At this point I'm hard pressed to believe it is the OS, but better understanding of what the app is trying to do could provide the correct evidence. – Zayne S Halsall Nov 20 '10 at 18:25
  • Well netcat doesn't really work because it occupies the port that it is listening on. I find it hard to see how a basic Ubuntu 64-bit 10.04 instance can be having such a huge problem starting glassfish using the standard sun java 6 installation. – orange80 Nov 20 '10 at 19:57
  • What I meant was *after* glassfish complains, then try netcat to prove immediately that *nothing* was actually listening on that port. It then points to a vagary of operation or configuration of the glassfish app, as opposed to then OS and network stack. And given the checking already done, speaking to the application developers is the logical next step, unless you're thinking of a doing a fresh install and trying again? ;) – Zayne S Halsall Nov 20 '10 at 21:32
  • Hey man, thanks for all your help. Because it's ec2, I can fairly easily fire up another instance from scratch and see if I run into the same problems. I'll report back once I've had time to check it out. – orange80 Nov 21 '10 at 05:19
  • I fired up another instance and had no problems setting up glassfish. I have no idea why that one instance didn't work. Maybe I changed something somewhere without realizing or something else went wrong. Anyways, problem solved. Thanks for helping :) – orange80 Nov 26 '10 at 07:01
0

I don't know why your hosts file was brought up, as these have nothing to do with each other.

This sounds to me (based on your revisions) that its not that something is currently listening on the port--but that you may have configured TWO things to listen on the same port in the same Glassfish instance. So if you start Glassfish and it tries to launch listeners on each of the ports, it will fail, and consequently shutdown the previously started services. So when you use netstat before and after, it reports correctly that nothing is currently listening on the ports.

Check that each of your listeners is on a different port using asadmin, and I'll bet you find that the port number is used twice. Or it could simply be that the initialization is causing two instances to start up simultaneously, which would give you the same behavior. Can you try configuring a brand new domain on different ports to see if you experience a similar issue?

Best of luck!

UPDATE: Try using the following to get all the listeners:

asadmin get server | grep -i listener
Andrew M.
  • 11,182
  • 2
  • 35
  • 29
  • Unfortunately the server refuses to start, I get this: # bin/asadmin get server | grep -i listener Remote server does not listen for requests on [localhost:4848]. Is the server up? – orange80 Nov 21 '10 at 05:42
  • Try `asadmin get server.*`. What you're seeing is probably `stderr`, not `stdout`--pipes will not pipe stderr unless instructed to do so. If you are still not getting anything, then it sounds like there are no configured listeners... which indicates even more issues with your install. – Andrew M. Nov 21 '10 at 15:46
0

same problem here. It is related to /etc/hosts i did a cat /etc/hostname and added this for 127.0.0.1

javadude
  • 239
  • 2
  • 6
  • 14
0

This error is thrown not only when the port is taken, but also erroneously when there is an error with the /etc/hosts file.

Make sure you have proper definitions for your hostname (have you changed it recently?) in the file, and that it is not missing. For example, if you just changed your hostname (either via hostname or in /etc/hostname) to server.example.com, then make sure that your /etc/hosts file has:

127.0.0.1 server.example.com localhost
Yuval A
  • 257
  • 1
  • 3
  • 11
0

Recently, I found that launching a server into VPC requires quite a bit of extra configuration (once off) to get started. A key clue:

 nslookup `hostname`

should work. If it doesn't work, that's likely to be the reason.

You can either: 1) launch a server into EC2-classic 2) configure your VPC / host to be able to run the command above successfully.

Hope that helps.

Paul Jowett
  • 101
  • 2