0

I am working with Spark and Cassandra and in general things are straight forward and working as intended; in particular the spark-shell and running .scala processes to get results.

I'm now looking at utilisation of the Spark Job Server; I have the Job Server up and running and working as expected for both the test items, as well as some initial, simple .scala developed.

However I now want to take one of the .scala programs that works in spark-shell and get it onto the Spark Job Server to access via that mechanism. The issue I have is that the Job Server doesn't seem to recognise the import statements around cassandra and fails to build (sbt compile; sbt package) a jar for upload to the Job Server.

At some level it just looks like I need the Job Server equivalent to the spark shell package switch (--packages datastax:spark-cassandra-connector:2.0.1-s_2.11) on the Spark Job Server so that import com.datastax.spark.connector._ and similar code in the .scala files will work.

Currently when I attempt to build (sbt complie) I get message such as: [error] /home/SparkCassandraTest.scala:10: object datastax is not a member of package com [error] import com.datastax.spark.connector._

I have added different items to the build.sbt file based on searches and message board advice; but no real change; if that is the answer I'm after what should be added to the base Job Server to enable that usage of the cassandra connector.

ChrisHDog
  • 4,473
  • 8
  • 51
  • 77
  • Do you want to run multiple programs that use Cassandra? Or only one? Did you try to pack your code into "fat jar" via `sbt assembly` & submit? – Alex Ott Dec 07 '17 at 09:30
  • not sure ... i just want to build a simple scala that connects to Cassandra and reads back some data so I can confirm it works and then think about what I want to do from there ... i'd want to create a jar that I can use the curl commands to load to the spark job server and then call them from there – ChrisHDog Dec 13 '17 at 04:54
  • I'm not familiar with Job Server unfortunately, but it looks like that your packaging instructions miss the cassandra dependency... – Alex Ott Dec 13 '17 at 07:49
  • That sounds right (i.e. that the cassandra dependency isn't being picked up); what do I need to do in intelliJ to make that happen? – ChrisHDog Dec 14 '17 at 00:26

1 Answers1

0

I think that you need spark-submit to do this. I am working with Spark and Cassandra also, but only since one month; so I've needed read a lot of information. I had compiled this info in a repository, maybe this could help you, however is an alpha version, sorry about that.

Juan Antonio Aguilar
  • 136
  • 1
  • 11