2

I want to write data in delta format to a hdfs which is accessible via webhdfs. Writing in parquet format works like charm but when I set the format to delta I get the following error

20/03/25 18:12:18 WARN DeltaLog: Failed to parse webhdfs://somethingsomething:50070/development/deltaTest/_delta_log/_last_checkpoint. This may happen if there was an error during read operation, or a file appears to be partial. Sleeping and trying again. org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: webhdfs

Code currently looks as follows:

val connectionString = s"webhdfs://somethingsomething:50070/development/deltaTest"
val conf = new SparkConf().setMaster("local[*]").setAppName("webhdfs test")

val spark = SparkSession.builder().config(conf).getOrCreate()
spark.sparkContext.setLogLevel("WARN")
import spark.implicits._
val data = spark.range(0,5)
data
  .write
  .format("delta")
  .mode(SaveMode.Overwrite)
  .save(connectionString)

Is it possible to do this with delta format?

user2811630
  • 445
  • 3
  • 11

0 Answers0