Stack Overflow
Stack Overflow

Questions tagged [spark-notebook]

The Spark Notebook is a web application enabling interactive and reproductible data analysis using Apache Spark from the browser

120 questions
0
votes
2 answers

Path contains empty name

I am running the below task to Upload Notebook to Databricks with Azure DevOps release pipeline: databricks workspace mkdirs //build databricks workspace import --language PYTHON --format SOURCE --overwrite…
user13557676
  • 1
0
votes
1 answer

Databricks Notebook - Microsoft Azure - Autocompletion is NOT working when attached to cluster

Databricks Notebook - Microsoft Azure - Autocompletion is NOT working when databricks notebook is attached to cluster. Can someone please share any pointers to fix this? [Edited] I have a couple of clusters, and i see this behaviour of autocomplete…
Raja
  • 513
  • 5
  • 18
0
votes
2 answers

Loading external jars into spark-notebook fails

I am trying to connect to redshift from notebook, so far i have done following - Configured metadata for the notebook "customDeps": [ "com.databricks:spark-redshift_2.10:3.0.0-preview1", "com.databricks:spark-avro_2.11:3.2.0", …
Guruprasad
  • 41
  • 6
0
votes
1 answer

Spark Notebook: Does GeoPointsChart accept a Dataframe?

I have a Dataframe which has two columns latitude and longitude. I passed that to GeoPointsChart. The output is "showing 1000 rows" but it isn't actually showing me anything. Has anyone faced the same issue? Is this a syntactical mistake?
Kirtiman Sinha
  • 843
  • 8
  • 19
0
votes
0 answers

Zeppelin doesn't work with external spark - Can not get or update progress

I've installed Zeppelin 0.6.2 and trying to get it working with external spark cluster (2.0.1). I have my Interpreter setting as follows: args master 121.201.~~.1~~:8080 spark.app.name Zeppelin spark.cores.max spark.executor.memory zeppelin.R.cmd…
Shengjie
  • 12,336
  • 29
  • 98
  • 139
0
votes
0 answers

How to do a simple table read, duplicate a whole row, and save it as a new table?

I am new to DataBricks and Apache Spark. I would like to use Python for my notebook. I am using data bricks and I can confirm I have created: A Cluster (test) along with a attached python based notebook (test-notebook) A Table from JSON File…
Techno04335
  • 1,365
  • 6
  • 22
  • 43
0
votes
1 answer

bluemix pyspark error occurred while calling jdbc

I'm using pySpark in the Spark service in Bluemix to transform something in my data and then write it in DashDB, also in Bluemix. But when I try to load data I receive the following error: Py4JJavaError: An error occurred while calling o111.jdbc. :…
Bruno Hirata
  • 21
  • 3
0
votes
1 answer

Automate zeppelin notebooks

Hi I just watched a databricks webinar where they are automating a databricks notebook and they have a dependent jar that the notebook is dependant on as well. Can you do this Zeppelin? Like calling the script in a notebook from a bash script. …
uh_big_mike_boi
  • 3,350
  • 4
  • 33
  • 64
0
votes
3 answers

unable to set up spark notebook getting the no such command error and permission denied errors

I am trying to run spark notebook on my machine and I followed the procedure mentioned in "spark notebook io". During this procedure, I am giving the command bin/spark-notebook, in the terminal, and I am getting Error: permission denied and when I…
kaladhar1991
  • 1
  • 1
0
votes
1 answer

Building spark notebook for windows with sbt

Can spark notebook be built with Windows ? I've checked out latest source and run via 'sbt run' but receive error : workspaces\spark-notebook-master\app\Global.scala:11: object share is not a member of package notebook [error] …
blue-sky
  • 51,962
  • 152
  • 427
  • 752
0
votes
0 answers

spark-notebook “Bad substitution” when submitting spark job to yarn-cluster

Similar to "Bad substitution" when submitting spark job to yarn-cluster I get the following when submitting job to yarn cluster 2016-02-25 19:49:11,029 INFO [Remote-akka.actor.default-dispatcher-4] (org.apache.spark.deploy.yarn.Client) -…
oluies
  • 17,694
  • 14
  • 74
  • 117
0
votes
2 answers

Spark: Exception in thread "main" akka.actor.ActorNotFound:

I am submitting my spark jobs from a local laptop to a remote standalone Spark cluster (spark://IP:7077). It is submitted successfully. However, I do not get any output and it fails after some time. When i check the workers on my cluster, I find the…
Spark User
  • 11
  • 5
0
votes
1 answer

Bluemix spark service error: java.net.NoRouteToHostException

Yesterday, this code ran ok. However today I get the following error message: df = sqlContext.read.jdbc( url="jdbc:db2://bluemix05.bluforcloud.com:50001/BLUDB:user=****;password=****;sslConnection=true;", …
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
0
votes
1 answer

Why does Spark Notebook in Hue report "Gateway timeout Error 504"?

I'm using Hadoop 2.2.6 mini cluster (1 Master and 3 slaves) with Ambari 2.1.0 and Hue 3.8.1, Ubuntu 12.04. Spark 1.2.1 (using Scala 2.10.3) was installed as part of the Ambari setup. I've installed Livy-Server to be able use Spark Notebook in Hue.…
USSR
  • 301
  • 1
  • 3
  • 22
-2
votes
1 answer

Broadcast dataframe is not removing shuffle phase (few rows of data)

I am learning spark using databricks community edition notebook. I have created sample data with few rows. As data is really small it should not have exchange phase in query plan. I tried broadcast too, still I see exchange phase. Do these…
user2895589
  • 1,010
  • 4
  • 20
  • 33
1 2 3 4 5 6 7
8