The EMS Server has its own built-in JNDI server. What you're actually doing when you connect is 1) querying the EMS:s JNDI server for a connection factory definition and then 2) creating a connection based on the returned factory. This is implied by the fact that you're using java.naming.provider.url.
Change the connection factory definition (factories.conf) on the EMS server for the connection factory you're using. The default definition for the default factories (e.g. QueueConnectionFactory) on a fresh install is "tcp://7222" which will be replaced by "tcp://hostname:7222" by the server when retrieved. You can change this definition to e.g. "tcp://hostname.myfqdn.com:7222" and things should work.
You could also bypass the JNDI server completely by creating a connection directly, but I wouldn't recommend this since the connection factory definition returned by the server may contain information about load balanced and fault tolerant pairs, SSL settings, or point to a completely different server pair etc. It also allows the EMS administrators to change the definition of connection factories without the clients having to change their code or even their configuration.