0

When we try to submit Spark job using ADF or Livy in HDInsight 3.6, Spark 2.1 with Secure transfer required "Enabled" in associated storage account. It is not submitting Spark job and we are unable to see Spark Job in YARN UI, but the same way we are able to submit, if we disable this Secure transfer required in associated storage account.

Please let us know how to Submit Spark job if Secure transfer required is "Enabled" in storage account.

wonea
  • 4,783
  • 17
  • 86
  • 139
Sudeep Singh Thakur
  • 79
  • 1
  • 2
  • 10

1 Answers1

0

I got the solution for this problem. Before creating HDinsight cluster keep ready below points. **

1 Create azure storage account and create container where you want to put your jar or Spark related code.
2 While Configuring HDInsight cluster you use the above created storage account and container.

**

Thanks, Sudeep

Sudeep Singh Thakur
  • 79
  • 1
  • 2
  • 10