0

I can't seem to find a way to start a shell using all the servers set up in conf/servers

Only found it possible to submit to cluster jobs using /bin/snappy-job.sh where I specify the lead location, but would like to try real time shell to perform some tests using the whole cluster

Thank you,

Saif

plamb
  • 5,636
  • 1
  • 18
  • 31
Saif
  • 57
  • 1
  • 7

1 Answers1

2

Please see this link. It tells how to start a spark-shell and connect it to snappy store.

http://snappydatainc.github.io/snappydata/connectingToCluster/#using-the-spark-shell-and-spark-submit

Essentially you need to provide the locator property and this locator is the same which you have used to start the snappy cluster.

$ bin/spark-shell --master local[*] --conf snappydata.store.locators=locatorhost:port --conf spark.ui.port=4041

Note that with the above a different compute cluster is created to run your program. The snappy cluster is not used for computation when you run your code from this shell. The required table definition and data is fetched in efficient fashion from the snappy store.

In future we might make this shell connect to the snappy cluster in such a way that it uses the snappy cluster itself as its compute cluster.

plamb
  • 5,636
  • 1
  • 18
  • 31
K Neeraj
  • 31
  • 2
  • Whilst this may theoretically answer the question, [it would be preferable](//meta.stackoverflow.com/q/8259) to include the essential parts of the answer here, and provide the link for reference. – Brian Ray Mar 22 '16 at 18:08
  • Thanks. So even if the --master is local[*], the shell connects to the spark driver (Job Server) and utilize the servers setup for the snappy cluster? Will any spark action utilize all conf/servers hosts for computation and data distribution? – Saif Mar 31 '16 at 15:31