0

I'm new to pyspark.I'm using python 3.5 & spark2.2.0 on my Ubuntu 16.0. I wrote following code to connect BigSQL using pyspark

from pyspark.sql.session import SparkSession
spark = SparkSession.builder.getOrCreate()

spark_train_df = spark.read.jdbc("jdbc:db2://my bigsq url :port number:sslConnection=true;sslTrustStoreLocation=ibm-truststore.jks;sslTrustStorePassword=*password123;","schema.Table Name",
             properties={"user": username, 
                      "password": password,
                      'driver' : 'com.ibm.db2.jcc.DB2Driver'}) # Trust store location is defined in .bashrc
spark_train_df.registerTempTable('data_table')

train_df = spark.sql('select * from data_table')

Also I have added my trust store & driver path in my .bashrc file But while running this code I'm getting error message

java.lang.ClassNotFoundException: com.ibm.db2.jcc.DB2Driver exception

Can you expert please guide me to solve this problem?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
Python Learner
  • 437
  • 2
  • 11
  • 28
  • Please edit your question to add these details (1) which Db2-client name is installed (2) which is the version/fixpack of that Db2-client (3) Are you able to connect to the Db2 database from the bash command line or any other jdbc application running on ubuntu. – mao Oct 13 '17 at 13:32
  • Sorry, I don't have much clue about DB2 Client. If you can kindly suggest me how to find those information then it will be helpful. As per as I know that latest DB2 client is being installed. Yes I'm able to connect BigSQL using python on Ubuntu – Python Learner Oct 13 '17 at 14:02
  • How do you execute the code? What's the command line for pyspark? – Jacek Laskowski Oct 13 '17 at 19:17
  • After writing the code in Jupyter note book I'm using jupyter's run command to execute the code – Python Learner Oct 14 '17 at 09:29

1 Answers1

2

You need to add the DB2 JDBC jars in your spark-submit, i.e., for postgres

spark-shell --master local[*] --packages  org.postgresql:postgresql:9.4.1207.jre7

or (or DB2)

spark-shell --master local[*] --jars /path/to/db2/jdbc/db2.jar
Zouzias
  • 2,330
  • 1
  • 22
  • 32