Apache Kylin Cube Building fails (When using mysql JDBC connector)

Question

Apache kylin fails with a MySQLSyntaxErrorException even though the tables which are marked as non existing in the error log exists. Cube build fails in first step "SQoop to flat hive table" I am using Apache Kylin 2.5 in cloudera cdh 5.7 sandbox running on a docker.

Kylin Error Log

java.io.IOException: OS command error exit with return code: 1, error message: SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/apache_kylin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/01/09 00:26:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.0
19/01/09 00:26:17 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/01/09 00:26:17 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
19/01/09 00:26:17 INFO manager.SqlManager: Using default fetchSize of 1000
19/01/09 00:26:17 INFO tool.CodeGenTool: Beginning code generation
19/01/09 00:26:18 INFO manager.SqlManager: Executing SQL statement: SELECT TABLE1.PID as TABLE1_PID FROM SPRINGDB.TABLE1 as TABLE1  INNER JOIN SPRINGDB.TABLE2 as TABLE2 ON TABLE1.PID = TABLE2.PID WHERE 1=1 AND  (1 = 0) 
19/01/09 00:26:18 ERROR manager.SqlManager: Error executing statement: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'SPRINGDB.TABLE1' doesn't exist
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'SPRINGDB.TABLE1' doesn't exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
    at com.mysql.jdbc.Util.getInstance(Util.java:360)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2030)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:777)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
    at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:253)
    at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:336)
    at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1858)
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1657)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
19/01/09 00:26:18 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1663)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

The command is: 
/usr/lib/sqoop/bin/sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true  -Dmapreduce.job.queuename=default --connect "jdbc:mysql://localhost:3306/springdb" --driver com.mysql.jdbc.Driver --username root --password cloudera --query "SELECT TABLE1.PID as TABLE1_PID FROM SPRINGDB.TABLE1 as TABLE1  INNER JOIN SPRINGDB.TABLE2 as TABLE2 ON TABLE1.PID = TABLE2.PID WHERE 1=1 AND \$CONDITIONS" --target-dir hdfs://quickstart.cloudera:8020/kylin/kylin_metadata/kylin-bac2bbda-f64d-4e90-0cac-beb2902412e4/kylin_intermediate_testcube2_01c6159a_68c8_6da9_a88f_7a3e34f140b1 --split-by TABLE1.PID --boundary-query "SELECT min(TABLE1.PID), max(TABLE1.PID) FROM "SPRINGDB".TABLE1 as TABLE1" --null-string '' --fields-terminated-by '|' --num-mappers 4
    at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:95)
    at org.apache.kylin.source.jdbc.CmdStep.sqoopFlatHiveTable(CmdStep.java:49)
    at org.apache.kylin.source.jdbc.CmdStep.doWork(CmdStep.java:60)
    at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
    at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
    at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
    at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

For anyone who's following up I solved the issue by having the names of database tables and fields in all uppercase. — Nissanka Seneviratne, Feb 06 '19 at 09:48

Xiaoxiang Yu · Answer 1 · 2019-01-14T14:44:40.100

1

Nissanka Seneviratne, I found your log indicated that sqoop cannot find your table "SPRINGDB.TABLE1", it looks like that is the cause. Please try to confirm that your table of specific database can be accessed from provided user/account (and host accessible).

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'SPRINGDB.TABLE1' doesn't exist

If you found it not the root cause, please reply me.

edited Jan 14 '19 at 14:44

answered Jan 14 '19 at 14:34

Xiaoxiang Yu

115
10

The table can be accessed from the provided user/account and the host is also accessible. – Nissanka Seneviratne Jan 16 '19 at 06:27

Apache Kylin Cube Building fails (When using mysql JDBC connector)

1 Answers1