0

I am looking for an interactive way to work with the spark-shell that would resemble rstudio. My use is simple - I want to write code in the editor and execute line-by-line in the terminal of spark shell. I have tried Jupyter notebooks and Zeppelin, but they seem to not yet be as ready as the Python backends. Currently I cut and paste every time, but that becomes tiring after a short while. Google searches haven't come up with anything I found useful.

DeanLa
  • 1,871
  • 3
  • 21
  • 37
  • Instead of writing in editor, why don't you write the code directly in spark-shell. Are you expecting suggestions of methods from a class in editor..? – Gokul Potluri Jul 05 '17 at 10:59
  • I want my code to be reusable and to be saved to file, later to be run as a whole script. – DeanLa Jul 05 '17 at 11:14
  • Did you check this https://stackoverflow.com/questions/44346776/how-to-run-scala-script-using-spark-submit-similarly-to-python-script/44347237#44347237 ? – eliasah Jul 05 '17 at 12:18

2 Answers2

0

You can use Jupyter notebook, But you need respective kernel to be installed and configured to Jupyter.
All available kernels for Jupyter

As you said spark-shell, I assume you are writing scala code.
Available kernels for scala are : sparkmagic and Jupyter-scala

As of now Jupyter will allow you to download the script as a file with .py extension
Changing its extension and compiling will be an overhead(even though it can reduce your work to some extent)

Gokul Potluri
  • 262
  • 4
  • 16
  • I have tries toree and and SparkNotebook. It seems as if notebooks are currently not what I'm looking for. – DeanLa Jul 05 '17 at 11:33
0

There two options:

  1. Using Scala Worksheets in Intellij IDEA. It can simultaneously display the result as you type.There is no need to copy paste the code every time. Refer to this question to set it up: How to setup Intellij 14 Scala Worksheet to run Spark
  2. Build your own environment by tweaking Atom text editor. There are various plugins in atom to include a terminal along with code editor. Install one of them and open spark REPL in the terminal. If you prefer, you can add a shortcut to run the selected code in the REPL from editor(like ctrl+enter in Rstudio) Refer: https://atom.io/packages/terminal-plus
vdep
  • 3,541
  • 4
  • 28
  • 54