-2

I installed "VScode, jdk 8, python 3.8 and databricks-connect==8.1.*" in Azure Windows Virtual Machine. After that I created a databricks cluster and configured Databricks-connect by using cmd. After setting all the path variables I executed following Java code:

'''

import java.util.ArrayList;
import java.util.List;
import java.sql.Date;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.*;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.Dataset;

public class test_{
    public static void main(String[] args) throws Exception {
        SparkSession spark = SparkSession.builder().appName("Temps Demo").config("spark.master", "local").getOrCreate();


        // Create a Spark DataFrame consisting of high and low temperatures
        // by airport code and date.
        StructType schema = new StructType(new StructField[] {
            new StructField("AirportCode", DataTypes.StringType, false, Metadata.empty()),
            new StructField("Date", DataTypes.DateType, false, Metadata.empty()),
            new StructField("TempHighF", DataTypes.IntegerType, false, Metadata.empty()),
            new StructField("TempLowF", DataTypes.IntegerType, false, Metadata.empty()),
        });
}}'''

========================================================= Inside VScode terminal I am getting below output '''

PS C:\Users\gaurav>  & 'c:\Users\gaurav\.vscode\extensions\vscjava.vscode-java-debug-0.35.0\scripts\launcher.bat' 'C:\Program Files\AdoptOpenJDK\jdk-11.0.11.9-hotspot\bin\java.exe' '-Dfile.encoding=UTF-8' '-cp' 'C:\Users\gaurav\AppData\Local\Temp\vscodesws_8933a\jdt_ws\jdt.ls-java-project\bin' 'App' 
    Exception in thread "main" java.lang.Error: Unresolved compilation problems: 
            SparkSession cannot be resolved to a type
            SparkSession cannot be resolved
            StructType cannot be resolved to a type
            StructType cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
    PS C:\Users\gaurav>  & 'c:\Users\gaurav\.vscode\extensions\vscjava.vscode-java-debug-0.35.0\scripts\launcher.bat' 'C:\Program Files\AdoptOpenJDK\jdk-11.0.11.9-hotspot\bin\java.exe' '-Dfile.encoding=UTF-8' '-cp' 'C:\Users\gaurav\AppData\Local\Temp\vscodesws_8933a\jdt_ws\jdt.ls-java-project\bin' 'test_'
    Exception in thread "main" java.lang.Error: Unresolved compilation problems: 
            SparkSession cannot be resolved to a type
            SparkSession cannot be resolved
            StructType cannot be resolved to a type
            StructType cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
    
            at test_.main(testjava.java:22)

'''

When I run "databricks-connect test" in cmd after configuring databricks connect. I get following output... [1]: https://i.stack.imgur.com/p5WG6.png

Abhay Pratap Singh
  • 1
  • 1

1 Answers1

0

org.apache.spark.sql.types.StructType is achievable in spark-catalyst, add both spark-catalyst.jar and spark-sql.jar to Referenced Libraries.

Besides, java extension needs jdk11 or recent version to support.

Above conditions met, no StructField cannot be resolved to a type shown in my project.

Molly Wang-MSFT
  • 7,943
  • 2
  • 9
  • 22