0

I am using windows machine and installed spark and scala for my learning. for spark-sql in need to process json input data. 

scala> sc
res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@7431f4b8

scala> import play.api.libs.json._
<console>:23: error: not found: value play
       import play.api.libs.json._
              ^

scala>

How can i import play api in my spark-shell commnad.

Learn Hadoop
  • 2,760
  • 8
  • 28
  • 60
  • 1
    You can create a build.sbt, which includes play as a dependency. And then load sbt console, and it should work. Check this: https://stackoverflow.com/questions/17430164/importing-jar-files-into-scala-environment – Shankar Shastri Sep 28 '18 at 04:44
  • @ShankarShastri. thanks a lot. in maven jar , we are providing org.ocpsoft.common common-api 1.0.5.Final in mvn. how can do provide the same entry in sbt file. – Learn Hadoop Sep 28 '18 at 04:48
  • Can you share the build.sbt, I can create an answer and post it back. – Shankar Shastri Sep 28 '18 at 06:40
  • You should have something called as jars directory in your spark installation path. In my Mac, its under SPARK_HOME/libexec directory. Can you copy the required jar in that directory and restart your spark-shell ? This should work – Constantine Sep 28 '18 at 07:12

1 Answers1

2

If you want to use other libraries while you are using spark-shell, you need to run spark-shell command with --jars and/or --packages. For example, to use play in your spark shell, run the following command;

spark-shell --packages "com.typesafe.play":"play_2.11":"2.6.19"

For more information, you can use spark-shell -h. I hope it helps!

ulubeyn
  • 2,761
  • 1
  • 18
  • 29
  • Thanks a lot ulubeyn. I think it is short term solution in spark-shell. How can i do permanent one like pom.xml – Learn Hadoop Sep 28 '18 at 07:46
  • I can recommend another way but I need to know one thing; do you want to use `spark-shell` or just `spark`? – ulubeyn Sep 28 '18 at 07:54
  • I recommend you to use SBT for Scala projects, because it is default build tool. Also, you can do lots of things with SBT files, like creating your own `SparkContext` as described [here](https://sanori.github.io/2017/06/Using-sbt-instead-of-spark-shell/). – ulubeyn Sep 28 '18 at 08:09
  • If you are developing a Spark application, it is better to use `build.sbt` which includes Spark and Play as a dependency, because you can make your own configurations, or create your jar to run repetitively in cloud/somewhere else. If you are just developing a simple Spark job, create a `.scala` file and run it with `spark-shell` specifying `play` as a dependency. – ulubeyn Sep 28 '18 at 08:21