0

I am trying to write a dataframe to a table:

spark.sql("CREATE DATABASE IF NOT EXISTS my_db")
spark.catalog.setCurrentDatabase("my_db")
dataFrame.write
        .format("csv")
        .mode(SaveMode.Overwrite)
        .bucketBy(5, "NAME", "DEPT")
        .sortBy("NAME", "DEPT")
        .saveAsTable("my_tbl")

but I get error:

java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

HADOOP_HOME is set to winutils location, and %HADOOP_HOME%\bin is added to PATH.

Dependencies are:

scalaVersion := "2.12.14"
val sparkVersion = "3.3.0"
useCoursier := false
val sparkDependencies = Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-avro" % sparkVersion
)
Mandroid
  • 6,200
  • 12
  • 64
  • 134

0 Answers0