Apache Spark fails to connect to MonetDB Cluster using JDBC Driver

Question

We have problems connecting to a MonetDB Cluster via Apache Spark using JDBC. Connection to a database that is not clustered works. But when we try to connect to a clustered MonetDB database via Apache Spark, it fails with a "unhandled result type error". The entire query is given below

We have tried plain JDBC to connect to a monetdb cluster and it works. Only Spark fails.

val v1 = hiveContext.load("jdbc",Map("url" -> "jdbc:monetdb://1.1.1.1/tpch1?user=monetdb&password=monetdb","dbtable" -> "(select count(*) from customer)v1"))
**java.sql.SQLException: node */tpch/1/monet returned unhandled result type**

java.sql.SQLException: node */tpch/2/monet returned unhandled result type
        at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2536)
        at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnection.java:2284)
        at nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508)
        at nl.cwi.monetdb.jdbc.MonetStatement.execute(MonetStatement.java:349)
        at nl.cwi.monetdb.jdbc.MonetPreparedStatement.<init>(MonetPreparedStatement.java:118)
        at nl.cwi.monetdb.jdbc.MonetConnection.prepareStatement(MonetConnection.java:901)
        at nl.cwi.monetdb.jdbc.MonetConnection.prepareStatement(MonetConnection.java:825)
        at org.apache.spark.sql.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:96)
        at org.apache.spark.sql.jdbc.JDBCRelation.<init>(JDBCRelation.scala:125)
        at org.apache.spark.sql.jdbc.DefaultSource.createRelation(JDBCRelation.scala:114)
        at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:290)
        at org.apache.spark.sql.SQLContext.load(SQLContext.scala:679)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:23)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30)
        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:32)
        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:34)
        at $iwC$$iwC$$iwC.<init>(<console>:36)
        at $iwC$$iwC.<init>(<console>:38)
        at $iwC.<init>(<console>:40)
        at <init>(<console>:42)
        at .<init>(<console>:46)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

RD 1429856293844: read final block: 67 bytes RX 1429856293845:
bwnthqobupI3CPY:merovingian:9:RIPEMD160,SHA256,SHA1,MD5:LIT:SHA512: RD 1429856293845: inserting prompt TD 1429856293846: write final block: 99 bytes TX 1429856293846: BIG:merovingian:{SHA256}7070477fd396595c929386453a32ac5059a9b76976e54dab3fbb7a8f5299bd25:sql:tpch1: RD 1429856293847: read final block: 0 bytes RX 1429856293847: RD 1429856293847: inserting prompt TD 1429856293847: write final block: 49 bytes TX 1429856293847: sSET TIME ZONE INTERVAL '+05:30' HOUR TO MINUTE ; RD 1429856293848: read final block: 3 bytes RX 1429856293848: &3

RD 1429856293848: inserting prompt TD 1429856293855: write final block: 15 bytes TX 1429856293855: Xreply_size 250 RD 1429856293855: read final block: 0 bytes RX 1429856293855: RD 1429856293855: inserting prompt TD 1429856293855: write final block: 68 bytes TX 1429856293855: sPREPARE SELECT * FROM (select count(*) from customer)v1 WHERE 1=0 ; RD 1429856293856: read final block: 52 bytes RX 1429856293856: !node */tpch/2/monet returned unhandled result type

RD 1429856293856: inserting prompt

Thanks for your help Hannes. I have updated the post with full stack trace. Only cluster with multiplex funnel is a problem. There is no problem connecting to a single node. — user2238704, Apr 22 '15 at 04:50
If you connect with the extra parameter debug=1 (e.g. in your case `jdbc:monetdb://1.1.1.1/tpch1?user=monetdb&password=monetdb&debug=1`, the JDBC driver will produce a .log file with some helpful debug info. Perhaps you could share this output as well. — Hannes Mühleisen, Apr 23 '15 at 08:45
I tried with debug=1. No log file got created. Where does it get created?. I searched for recently created files after i ran the Spark job. I looked in the same directory as the JDBC folder, MonetDB folder as well as var/logs — user2238704, Apr 23 '15 at 12:41
Not sure where this file ends up in your spark setup, but the filename is `monet_1429855366161.log`, where the number is a timestamp. Also a correction, the parameter value should be `true`, not `1`. So `debug=true`. — Hannes Mühleisen, Apr 24 '15 at 06:05

score 0 · Answer 1 · answered Apr 24 '15 at 09:02

0

The problem lies not with JDBC, but with the merovingian funnel. It seems the funnel does not like the statement PREPARE SELECT * FROM (select count(*) from customer)v1 WHERE 1=0 ; See whether you can keep your application from using prepared statements. Feel free to file a bug report on this at http://bugs.monetdb.org

answered Apr 24 '15 at 09:02

Hannes Mühleisen

2,542
11
13

Its not we, but Spark that prepares the statement and sends it. So we cant do anything. Can you please let me know if this will be fixed ? – user2238704 Apr 27 '15 at 09:10

Apache Spark fails to connect to MonetDB Cluster using JDBC Driver

1 Answers1