7

I have a regular sbt scala spark project and would like to integrate with zeppelin. As you can imagine the project not only contains a single big file but rather some nice classes to modularize functionality.

I is unclear to me how this could translate to zeppelin at best in a way where both world are integrated. For now I only managed to copy past let all the code into a single zeppelin notebook which is not really what I would like to achieve.

edit

Zeppelin works well when code is written in a single notebook and all code fits into this notebook. However, when the code gets more complex and I would want to encapsulate (reusable) modules as a class (in a separate file). Can I simply open a direction in zeppelin and then have regular Scala import statements work fine (like in Jupiter with python)?

.
├── README.md
├── build.sbt
├── project
│   ├── build.properties
│   ├── plugins.sbt
├── scalastyle-config.xml
├── src
│   ├── main
│   │   ├── resources
│   │   │   ├── log4j.properties
│   │   │   └── zcta510.csv
│   │   └── scala
│   │       └── myOrg
│   │           ├── MainApplication.scala
│   │           ├── moduleAReferencedinMain.scala
│   │           └── bigModuleBAllSubPartsReferencedInMainViaImportStatements
│   │               ├── B_1.scala
│   │               ├── B_2.scala
│   │               └── B_3.scala
│   └── test
│       ├── resources
│       │   └── zcta510.csv
│       ├── scala
│       │   └── myOrg
│       │       └── ModuleATest.scala
└── version.sbt

Also this is related to https://forums.databricks.com/questions/1861/importing-functions-from-other-notebooks.html

MichaelChirico
  • 33,841
  • 14
  • 113
  • 198
Georg Heiler
  • 16,916
  • 36
  • 162
  • 292
  • If you are trying to write spark application (single jar), you can submit via `SPARK_SUBMIT` option in `$ZEPPELIN_HOME/conf/zeppelin-env.sh`. See http://zeppelin.apache.org/docs/0.7.1/interpreter/spark.html#1-export-spark_home – 1ambda May 05 '17 at 05:05
  • I couldn't fully understand what exactly you want to do. Could you explain more? – 1ambda May 05 '17 at 05:06
  • Please see the edit. But I want to produce modular code and do not see how this concept of modularity fits into Zeppelin. Additionally, It would be great if the world of Zeppelin could bridge nicely with the original set Scala project. – Georg Heiler May 05 '17 at 05:15
  • 1
    You can package your modules into jars and put them in a binary repository such as maven or Artifactory if you want. Then you can use `z.addRepo()` and `z.load()` to load your dependencies into another notebook before importing your classes and using them. See http://zeppelin.apache.org/docs/0.7.1/interpreter/spark.html#3-dynamic-dependency-loading-via-sparkdep-interpreter – snark May 11 '17 at 08:34
  • Is there a more dynamic option, i.e. like compile depend on library project? – Georg Heiler May 11 '17 at 08:36
  • It seems that if you do what I just said your libraries won't be accessible from `%spark.sql` so they recommend fiddling with the interpreter settings instead. See http://zeppelin.apache.org/docs/0.7.1/manual/dependencymanagement.html – snark May 11 '17 at 08:38

0 Answers0