How to set streaming app checkpointing to Azure storage?

Question

I am trying set checkpointing for spark streaming application to Azure storage. I was using S3 and the code was working fine.

Here is the latest code of how I set checkpointing to Azure.

sc.hadoopConfiguration
      .set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
    sc.hadoopConfiguration
      .set(
        "fs.azure.account.key.[name].blob.core.windows.net",
        [key]
      )
    ssc.checkpoint(
      "https://[name].blob.core.windows.net/[blob]")

Here is the error message that I am getting when starting. Exception in thread "main" java.io.IOException: No FileSystem for scheme: https

score 0 · Accepted Answer · answered Aug 20 '19 at 18:15

0

See here - it's for databricks but should still apply.

val df = spark.read.parquet("wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<directory-name>")

==> So, use wasbs instead of https

answered Aug 20 '19 at 18:15

silent

14,494
4
46
86

Exception in thread "main" java.io.IOException: No FileSystem for scheme: wasbs – MarkZ Aug 21 '19 at 04:23
This is the new error that I am getting after changing `https` to `wasbs`. – MarkZ Aug 21 '19 at 04:24

How to set streaming app checkpointing to Azure storage?

1 Answers1