0

I had a td-agent sending messages to a Kafka queue with no problem. But now I get the following error. I restarted the agent with no luck. I flushed the buffer manually with no luck.

Any advice on what is happeing is welcome

2019-03-20 16:44:06 +0100 [warn]: #0 Send exception occurred: Failed to send messages to test-250-events/0 2019-03-20 16:44:06 +0100 [warn]: #0 Exception Backtrace : /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ruby-kafka-0.6.8/lib/kafka/producer.rb:330:in deliver_messages_with_retries' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ruby-kafka-0.6.8/lib/kafka/producer.rb:238:inblock in deliver_messages' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ruby-kafka-0.6.8/lib/kafka/instrumenter.rb:23:in instrument' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ruby-kafka-0.6.8/lib/kafka/producer.rb:231:indeliver_messages' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/out_kafka_buffered.rb:281:in deliver_messages' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/out_kafka_buffered.rb:344:inwrite' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/compat/output.rb:131:in write' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1111:intry_flush' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1394:in flush_thread_run' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:445:inblock (2 levels) in start' /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2019-03-20 16:44:06 +0100 [info]: #0 initialized kafka producer: kafka 2019-03-20 16:44:06 +0100 [warn]: #0 failed to flush the buffer. retry_time=5 next_retry_seconds=2019-03-20 16:44:05 +0100 chunk="58471660306e5c7712d0c2ff1a546048" error_class=Kafka::DeliveryFailed error="Failed to send messages to test-250-events/0" 2019-03-20 16:44:06 +0100 [warn]: #0 suppressed same stacktrace

There is connectivity with Kafka cluster from the server:

user@server:~$ telnet digital.servicebus.windows.net 9093 Trying 13.69.64.0... Connected to digital-dev-01.servicebus.windows.net. Escape character is '^]'.

Thank you in advance

Marcel
  • 21
  • 5

1 Answers1

0

We experience exactly the same issue. Running fluentd with Kafka output in Kubernetes cluster in Azure North Europe.

Around 1PM March 19th it just stopped working from multiple clusters.

Edit: We found a solution on our end. Basically make sure that you have a later version of ruby-kafka and/or fluent-kafka-plugin

We achieved that by having this in our Kubernetes manifest.

containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.3.3-debian-kafka-1.2 command: ["/bin/sh"] args: ["-c", "cp /fluent-config/*.conf /fluentd/etc/ && gem install fluent-plugin-detect-exceptions && gem install fluent-plugin-prometheus && gem update fluent-plugin-kafka &&/fluentd/entrypoint.sh"]

Mats Iremark
  • 93
  • 6