Spark - Adding JDBC Driver JAR to Google Dataproc

Question

I am trying to write via JDBC:

df.write.jdbc("jdbc:postgresql://123.123.123.123:5432/myDatabase", "myTable", props)

The Spark docs explain that the configuration option spark.driver.extraClassPath cannot be used to add JDBC Driver JARs if running in client mode (which is the mode Dataproc runs in) since the JVM has already been started.

I tried adding the JAR path in Dataproc's submit command:

gcloud beta dataproc jobs submit spark ... 
     --jars file:///home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar

I also added the command to load the driver:

  Class.forName("org.postgresql.Driver")

But I still get the error:

java.sql.SQLException: No suitable driver found for jdbc:postgresql://123.123.123.123:5432/myDatabase

zero323 · Accepted Answer · 2015-10-05T22:03:35.433

4

From my experience adding driver to the properties usually solves the problem:

props.put("driver", "org.postgresql.Driver")
db.write.jdbc(url, table, props)

edited Oct 05 '15 at 22:03

answered Oct 05 '15 at 21:53

zero323

322,348
103
959
935

Works great. Should be added to docs. (note: the code I have in my question has not changed so that may or may not be necessary) – BAR Oct 05 '15 at 23:17
Funny thing is it is not always necessary but I cannot figure out what exactly makes a difference. – zero323 Oct 05 '15 at 23:29
complexity will do that to any system ;) – BAR Oct 05 '15 at 23:31

score 2 · Answer 2 · answered Oct 05 '15 at 22:20

You may want to try adding --driver-class-path to the very end of your command arguments:

gcloud beta dataproc jobs submit spark ... 
    --jars file:///home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar \
    --driver-class-path /home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar

Another approach if you're staging the jarfile onto the cluster before the job anyway is to dump the jarfile you need into /usr/lib/hadoop/lib/ where it should automatically be part of the driver classpath for both Hadoop and Spark jobs.

Putting the driver-class-path at the end did not work. – BAR Oct 05 '15 at 22:57 — BAR, Oct 05 '15 at 22:57

score 0 · Answer 3 · answered Apr 20 '18 at 04:48

You can add jar (from --jars argument) to Spark Driver class-path using --properties argument when submitting Spark job through Dataproc:

$ gcloud dataproc jobs submit spark ... \
    --jars=gs://<BUCKET>/<DIRECTORIES>/<JAR_NAME> \
    --properties=spark.driver.extraClassPath=<JAR_NAME>

Spark - Adding JDBC Driver JAR to Google Dataproc

3 Answers3

Linked