I'm using Spark Job Server 0.6.2 with Spark 1.6.0 and in some of job submit attempts I get the following exception:
[ERROR] 2016-11-16 08:01:59,595 spark.jobserver.context.DefaultSparkContextFactory$$anon$1 logError - Error initializing…
I am using DSE Spark Job Server. The task I am attempting to accomplish is as follows:
The spark job which i have created in Java is expected to fetch some datas from cassandra db and this will be deployed in DSE Analytics cluster.
The code as…
My use case is as follows. I need to be able to call java methods from within python code
from py spark this seems to be very easy
I start the py spark like this
./pyspark --driver-class-path /path/to/app.jar
and from pyspark shell do…
I have a job that loads a data file from a different location each time. I'd like to submit the same job JAR and just pass a different location to it using the Config.java parameter of the runJavaJob() API.
I do not see a way to pass key/value…
Is there any specific Spark Job Server version matching with Spark 1.6.0 ?
As per the version information in https://github.com/spark-jobserver/spark-jobserver, I see SJS is available only for 1.6.1 not for 1.6.0.
Our CloudEra hosted Spark is…
I have a long running spark context via spark job server. The batch jobs are triggered periodically,
In some scenario the batch job got exception with following stack trace.
There is not much clue with the stack trace to see where the exception…
There is a lot I clearly don't understand about Spark, Spark Jobserver, and Mesosphere's DC/OS. But I very much like the Jobserver project, and also very much like our DC/OS cluster, and would really like to get them running together.
Throwing the…
I'm using Spark job-server for working with job management. I need to create 10 jobs e.g.. I can create 10 separate jars for it and call it next way:
curl -d ""…
Am trying to JAR a simple scala application which make use of SparlCSV and spark sql to create a Data frame of the CSV file stored in HDFS and then just make a simple query to return the Max and Min of specific column in CSV file.
I am getting error…
we have installed a spark-jobserver that is launched on a spark cluster using server_start.sh, however no matter how we go about it we cannot make it launch on multiple workers. We can manage to get it to run with several cores and more memory, but…
My project is implement a interaction query for user to discover that data. Like we have a list of columns user can choose then user add to list and press view data. The current data store in Cassandra and we use Spark SQL to query from it.
The Data…
we are working on launching a spark-jobserver on a spark cluster.
To deploy the server we follow the documentation at github.com/spark-jobserver/spark-jobserver#deployment. We deploy with by running ./server_deploy.sh local.
I've uploaded our…
getting started with spark I would like to know how to flatmap or explode a dataframe.
It was created using df.groupBy("columName").count and has the following structure if I collect it:
[[Key1, count], [Key2, count2]]
But I would rather like to…
I have issue while trying to start the job server. Im getting the below error,
./server_start.sh
./server_start.sh: line 41: kill: (113865) - No such process
Please help me on this.
I need to run an aggregation Spark job using spark-jobserver using low-latency contexts. I have this Scala runner to run a job on using a Java method from a Java class.
object AggregationRunner extends SparkJob {
def main(args: Array[String]) {
…
