Can't create table on apache spark ThriftServer through Spark Java API

Question

I have a simple Spark API java program described as follows:

SparkSession spark = SparkSession
            .builder()
            .appName("spark")
            .master("local")
            .getOrCreate();

ArrayList<TagClass> tagClass = new ArrayList<>();
TagClass tagClass1 = new TagClass();
tagClass1.setId(13);
tagClass.add(tagClass1);
Dataset<Row> rowDataset = spark.createDataFrame(tagClass,TagClass.class);

rowDataset.createOrReplaceTempView("TestTable");

rowDataset.write().option("createTableColumnTypes", "id INTEGER")
    .format("jdbc")
        .option("url", "jdbc:hive2://192.***.***.1:10000")
        .option("dbtable", "TestTable")
        .save();

this program tries to insert some data into a apache spark thriftserver table (hive).

When i run it on IntelliJ i get the following:

Exception in thread "main" java.sql.SQLException: org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'CREATE TABLE TestTable ("id"'(line 1, pos 27) 
== SQL ==
CREATE TABLE TabelaTeste2 ("id" int )
---------------------------^^^

Checking the thriftserver instance that is running on my windows10 machine (on cmd as superuser(admin)) shows me this error:

ERROR SparkExecuteStatementOperation: Error executing query, currentState RUNNING,
org.apache.spark.sql.AnalysisException: Table or view not found: TestTable;

followed by the same error that was displayed on IntelliJ.

Beeline connects just fine. On Beeline i'm able to select, insert on and create tables on the same instance of the thriftserver that i'm trying to operate with my java code.

Testing the SQL code provided by the error message on Beeline (CREATE TABLE TestTable ("id" int )), also results on the same error showed by both (IntelliJ and ThriftServer), but removing the quotes ("") from the SQL (CREATE TABLE TestTable (id int )) on beeline works just fine.

I don't know how to specify a "no quotes" behaviour on the SQL that sparks generates nor do i think that this is the issue, since this SQL code is generated by, i'm assuming, the .format("jdbc") option.

WRAP UP:

->I'm trying to create a table on my spark thriftserver programmatically, instead of doing so with beeline and i'm getting the above mentioned errors.

So, why am i getting this errors, what am i doing wrong?

My POM is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"


     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>spark-cassandra</groupId>
<artifactId>Sparkssandra</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive-thriftserver -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive-thriftserver_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc -->
    <dependency>
        <groupId>org.apache.hive</groupId>
        <artifactId>hive-jdbc</artifactId>
        <version>2.2.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector -->
    <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector_2.11</artifactId>
        <version>2.0.7</version>
    </dependency>

Thanks in advance!

EDIT:

I've removed the quotes from the SQL expression generated by the mettod createTable of JdbcUtils$ class and the table was created successfully, but no data was stored on it, instead, this error message appears on IntelliJ:

ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.sql.SQLException: Method not supported

Any clues?

Can't create table on apache spark ThriftServer through Spark Java API

0 Answers0