1

We were trying to use S3 for Flink backend state and checkpoints and used a bucket in Frankfurt (V4 authentication) It gave the error I posted here (Cannot access S3 bucket with Hadoop) and it was due to hadoop. However hadoop 2.8.0 works but there is no Flink support yet.

Guess my question is when will Flink offer a version based on hadoop 2.8.0?

razvan
  • 559
  • 7
  • 23

2 Answers2

1

Flink will probably offer a Hadoop 2.8.0 version once that Hadoop version is released.

In the meantime, you can build Flink yourself with a custom Hadoop version:

mvn clean install -DskipTests -Dhadoop.version=2.8.0

Robert Metzger
  • 4,452
  • 23
  • 50
1

Hadoop 2.7.x does work with Frankfurt and v4 API endpoints. If you are having problems, check your joda-time version as an odd combination of old joda-time JARs and Java versions causes AWS to get the wrong-formatted timestamp, which it then rejects in the ubiquitous "bad auth" message.

stevel
  • 12,567
  • 1
  • 39
  • 50
  • I used hadoop 2.7.3 and got that error, just copied the AWS, http* and jackson libs from share/hadoop/hdfs/lib to share/hadoop/common/lib as it was complaining about not finding them. – razvan Apr 28 '17 at 15:26
  • With hadoop 2.8 I did the same and it worked the first time. – razvan Apr 28 '17 at 15:27