0

I'm trying to access a wasb(Azure blob storage) file in Spark and need to specify the account key.

How do I specify the account in the spark-env.sh file?

fs.azure.account.key.test.blob.core.windows.net 
   EC5sNg3qGN20qqyyr2W1xUo5qApbi/zxkmHMo5JjoMBmuNTxGNz+/sF9zPOuYA==

WHen I try this it throws the following error

fs.azure.account.key.test.blob.core.windows.net: command not found
himanshuIIITian
  • 5,985
  • 6
  • 50
  • 70
user1050619
  • 19,822
  • 85
  • 237
  • 413

1 Answers1

0

From your description, it is not clear that the Spark you used is either on Azure or on local.

For Spark running on local, refer this blog post which introduces how to access Azure Blob Storage from Spark. The key is that you need to configure Azure Storage account as HDFS-compatible storage in core-site.xml file and add two jars hadoop-azure & azure-storage to your classpath for accessing HDFS via the protocol wasb[s].

For Spark running on Azure, the difference is just only access HDFS with wasb, all configurations have been done by Azure when creating HDInsight cluster with Spark.

AshokPeddakotla
  • 1,010
  • 7
  • 18
  • Thanks, I'm using spark local and I added a core site xml in my spark/conf directory and added the key for my blob storage as given here- https://blogs.msdn.microsoft.com/arsen/2016/07/13/accessing-azure-storage-blobs-from-spark-1-6-that-is-running-locally/ – user1050619 Sep 26 '17 at 10:07