1

We get an exception when setting up flink on Azure HDInsights cluster.

./bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m

Throws:

org.apache.flink.client.deployment.ClusterDeploymentException: 
Couldn't deploy Yarn session cluster

Caused by: 

Failing this attempt.Diagnostics: 
[2018-10-24 00:41:17.703]Resource wasb://../.flink/application_1539730571763_0057/
application_1539730571763_0057-flink-conf.yaml8158650202504017094.tmp 
changed on src filesystem (expected 1540341676000, was 1540341677000

java.io.IOException: 
    at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:273)

It seems to be because wasb blob storage is not keeping original timestamps for copied files breaking HDFS API abstraction on top of wasb. Any workarounds for this?

Only other thread I could find was Oozie/yarn: resource changed on src filesystem.

Fakrudeen
  • 5,778
  • 7
  • 44
  • 70

1 Answers1

0

We worked with MS Azure support and they confirmed that it only works with Flink 1.4.x and not 1.5.x or 1.6.x.

We downgraded it to 1.4.x and it is working fine now in Azure cluster.

Fakrudeen
  • 5,778
  • 7
  • 44
  • 70