0

I am trying to setup Spark standalone cluster in Azure cloud VM .Spark 2.2 setup is done. if I start the master (start-master.sh) , am able to see the master URL in web ui. But that spark master url has host name of that VM not IP address. VM IP has private & public , I am trying to configure master url with publilc IP so that other client can access.

I hope if the Spark master contains IP address instead of host name then my application access the master. I tried followed few steps mentioned in online forms , but nothing is working

option : 1 i created spark-env.sh under sbin folder and added SPARK_MASTER_HOST=x.x.x.x, but getting the same problem.

option : 2

while starting master i have passed host name as argument, spark-master.sh -h x.x.x.x. but same error

Error:

 your platform... using builtin-java classes where applicable
18/04/01 19:39:12 INFO SecurityManager: Changing view acls to: root
18/04/01 19:39:12 INFO SecurityManager: Changing modify acls to: root
18/04/01 19:39:12 INFO SecurityManager: Changing view acls groups to:
18/04/01 19:39:12 INFO SecurityManager: Changing modify acls groups to:
18/04/01 19:39:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7081. Attempting port 7082.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7082. Attempting port 7083.
 Attempting port 7093.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'sparkMaster' failed after 16 retries (starting from 7077)! Consider explicitly setting the appropriate port for the service 'sparkMaster' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481)
@@@

How to change spark master url ?

Gnana
  • 2,130
  • 5
  • 26
  • 57
  • You are certainly misreading the error message. This error says that master is failing to start, not client failing to connect. Seems that you're already having many **running** master instances. Try stopping all of them and retrying to launch master. If you use a host name/interface that the client can connect on, your driver should be able to connect. – ernest_k Apr 02 '18 at 06:31
  • I am able to understand that the issue in master startup. I ensured that no master is running before i start the master. my question here is how to change the master url with IP address . so that my client can access. I have a workaround for this. but i need a correct fix – Gnana Apr 02 '18 at 15:13
  • Is your scala client on same VM or different. If different, hope yo would have done proper settings to [expose ports](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/nsg-quickstart-portal). If done correct, you should be able to access the port with the public IP of master from the client VM. All above assuming your master has started well (as in the initial part of your Q) – sujit Apr 04 '18 at 06:49
  • Scala is running in same VM. Scala version : 2.11.8. But Master is not getting started . I don't know why ? – Gnana Apr 10 '18 at 04:58

0 Answers0