We have hadoop implemented on linux flatform. We use scala spark to develop models using spark machine learning libraries. I just use notepad++ and create *.scala file and execute them on data nodes. I want to know can I use eclipce or Intellij IDE from my local machine (windows machine) and connect to spark to test scala spark scripts.
Asked
Active
Viewed 120 times
1 Answers
0
You can use browser based notebooks like ipython or specifically jupyter (because it supports Scala) for testing your scripts. However you need to connect it to Hadoop Spark as explained in: http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/
abhiieor
- 3,132
- 4
- 30
- 47
-
Thank you very much for the reply. I read something called sparkmagic to connect spark from client machine through jupyter. And can we compile *.scala to jar file using jupyter. – Hasitha Indika Arumawadu Apr 21 '17 at 04:05