5

I'm using Kafka connect HDFS. When I'm trying to run my connector I'm got the following exception:

ERROR Failed creating a WAL Writer: Failed to create file[/path/log] for [DFSClient_NONMAPREDUCE_208312334_41] for client [IP] because this file is already beign create by [DFSClient_NONMAPREDUCE_165323242_41]

Any suggestions please?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Ya Ko
  • 509
  • 2
  • 4
  • 19

2 Answers2

0

As per further testing, upgrading the version to 2.7.7 didn't help. We still ran into WAL issue. We ended using a different sink s3SinkConnector.

Oliver Brylle Majaba
  • 31
  • 3
  • I think you need to describe more details of what you did and at least one reason for doing only that. Like the command to clean up. – Felipe Jun 02 '21 at 05:59
  • As per further testing, upgrading the version to 2.7.7 didn't help. We still ran into WAL issue. We ended using a different sink s3SinkConnector. – Oliver Brylle Majaba Sep 20 '22 at 14:41
0

Aside from the workaround to cleanup the wal logs and connector restart. We are also testing on upgrading the hadoop local version 2.73 to 2.7.7 and the kafka-hdfs version to 5.5 of our sink connector from Its currently on testing and so far the wal issue is not recoccuring.

Oliver Brylle Majaba
  • 31
  • 3