I need a ssh tunnel from my local computer running Windows 10 to an AWS EC2 host in order to monitor a java process via Java Mission Control. I followed countless guides for this, e.g. http://issamben.com/how-to-monitor-remote-jvm-over-ssh/. The parameters I use are kind of a combination of all these guides.The Java process on the host runs on Java 8 using the following jvm parameters:
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=9405
-Djava.rmi.server.hostname=10.0.1.1
-Dcom.sun.management.jmxremote.rmi.port=9415
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
I check whether the process is really listening on the JMX port by
nc -z localhost 9405
which tells me
Connection to localhost 9405 port [tcp/*] succeeded!
So far so good. Now let's go into the tunnel. I'm using the test-connection-feature of the JMC to check if the tunnel is working:
The following command is used to establish the tunnel:
ssh -l ec2-user -i .\ec2key.pem 46.123.23.231 -v -N -L localhost:9405:10.0.1.2:9405
Naturally 46.123.23.231 is the public IP of the EC2-instance. 10.0.1.2 is the internal IP of the EC2-instance in my VPC. I can successfully connect via ssh to the instance using just the first part of the command before '-v' or if I just leave out the '-N'. So my security-group seems to be configured as needed.
Let me show you the output of ssh:
debug1: Reading configuration data C:\\Users\\User/.ssh/config
debug1: Connecting to 46.123.23.231 [46.123.23.231] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file .\\ec2key.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file .\\ec2key.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 46.123.23.231:22 as 'ec2-user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:fkfgiekugo2iskelgioxq9itlkcs/KVISmiofjsaFSADf
debug1: Host '46.123.23.231' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\User/.ssh/known_hosts:7
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: .\\ec2key.pem
debug1: Authentication succeeded (publickey).
Authenticated to 46.123.23.231 ([46.123.23.231]:22).
debug1: Local connections to localhost:9405 forwarded to remote address 10.0.1.2:9405
debug1: Local forwarding listening on ::1 port 9405.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 9405.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
Hitting the Test-Connection-button in JMC yields:
debug1: Connection to port 9405 forwarding to 10.0.1.2 port 9405 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 9405 for 10.0.1.2 port 9405, connect from 127.0.0.1 port 53929 to 127.0.0.1 port 9405, nchannels 3
debug1: Connection to port 9405 forwarding to 10.0.1.2 port 9405 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 9405 for 10.0.1.2 port 9405, connect from 127.0.0.1 port 53931 to 127.0.0.1 port 9405, nchannels 3
debug1: Connection to port 9405 forwarding to 10.0.1.2 port 9405 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 9405 for 10.0.1.2 port 9405, connect from 127.0.0.1 port 53934 to 127.0.0.1 port 9405, nchannels 3
And JMC tells me 'Unable to connect' wihtout any specific error message. I configured the ssh-daemon on the host with
AllowTcpForwarding yes
I also tried to use a custom JMX service URL:
service:jmx:rmi://localhost:9415/jndi/rmi://localhost:9405/jmxrmi
What am I missing?
EDIT: Some websites suggest that the connection jmc uses opens some random ports and that's why it can't be used through frirewalls. So I tried to mimick the out-of-the-box-management like suggested here: https://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html.
So success there either.
