Error running introductory example of scala-xml

Question

I'm trying to run the simple example provided on the README of scala-xml, but the code won't run:

import org.apache.spark.sql.SQLContext

val sqlContext = new SQLContext(sc)
val df = sqlContext.read
    .format("com.databricks.spark.xml")
    .option("rowTag", "book")
    .load("books.xml")

(copy-pasted from the README; books.xml is indeed in the local directory)

This gives me error:

Name: Compile Error

Message: :1: error: illegal start of definition
.format("com.databricks.spark.xml")

^
StackTrace:

I'm running this from a Jupyter notebook with Spark/Scala kernel.

I'm sure there's a simple mistake, but I'm brand new to Scala/Spark.

Version info:

Spark: 2.0.1
Scala: 2.11.8

@Reactormonk `which scala` suggests the version is `scala-2.11.8` — MichaelChirico, Jul 24 '17 at 22:51
@Reactormonk running from one line gives a different error: "Failed to find data source: com.databricks.spark.xml. "... i don't see anything else on the README suggesting I need to do something else to import from databricks? — MichaelChirico, Jul 24 '17 at 22:56
have you added the library dependencies ??? com.databricks:spark-xml_2.11:0.4.1 ?? — Nayan Sharma, Jul 25 '17 at 07:59

score 0 · Answer 1 · edited Jul 25 '17 at 09:08

you can add packages to the Spark using the --packages command line option

As per the comment and question, you should try running the code in one line it will solve your problem of "error: illegal start of definition"

val df = spark.read.format("com.databricks.spark.xml").option("rowTag", "book").load("book.xml")

Next for "Failed to find data source: com.databricks.spark.xml. "

try adding the library dependency/package "com.databricks:spark-xml_2.11:0.4.1 "

spark-shell --packages com.databricks:spark-xml_2.11:0.4.1

val df = spark.read.format("com.databricks.spark.xml").option("rowTag", "book").load("book.xml")



df.show
+-----+--------------------+--------------------+---------------+-----+------------+--------------------+
|  _id|              author|         description|          genre|price|publish_date|               title|
+-----+--------------------+--------------------+---------------+-----+------------+--------------------+
|bk101|Gambardella, Matthew|An in-depth look ...|       Computer|44.95|  2000-10-01|XML Developer's G...|
|bk102|          Ralls, Kim|A former architec...|        Fantasy| 5.95|  2000-12-16|       Midnight Rain|
|bk103|         Corets, Eva|After the collaps...|        Fantasy| 5.95|  2000-11-17|     Maeve Ascendant|

How can I run this from a Jupyter notebook? i.e., how can I add command-line options to a Scala kernel in Jupyter? — MichaelChirico, Jul 25 '17 at 16:25

Error running introductory example of scala-xml

1 Answers1