Unable to send data from fluentd to Amazon Hosted Kafka

Question

I created an AWS MSK cluster, that use SASL/SCRAM Authentication as mentioned in the docs.

I am trying to send logs from fluentd to this AWS MSK using the fluentd-plugin-kafka

Below is the output plugin configuration I used

<match  **>
  @type kafka2

  # list of seed brokers
  brokers "b-1.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096,b-2.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096"
  sasl_over_ssl false
  username "<myusername>"
  password "********"
  scram_mechanism "sha256"

  # topic settings
  default_topic messages
  topic_key kubernetes

  get_kafka_client_log true

  <format>
      @type  json
  </format>

  # Optional. See https://docs.fluentd.org/v/1.0/configuration/inject-section
  <inject>
    tag_key tag
    time_key time
  </inject>

    # See fluentd document for buffer related parameters: https://docs.fluentd.org/v/1.0/configuration/buffer-section
    # Buffer chunk key should be same with topic_key. If value is not found in the record, default_topic is used.
  <buffer topic>
    flush_interval 10s
  </buffer>

    # ruby-kafka producer options
  max_send_retries 5

</match>

The fluentd plugin fails to communicate with the seed brokers and run into timeout errors as shown below

2021-05-26 15:59:00 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-05-26 15:59:01 +0000 chunk="5c33db6ccf6627f81dd216edd689b9ca" error_class=Kafka::ConnectionError error="Could not connect to any of the seed brokers:\n- kafka://b-2.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096: Connection timed out\n- kafka://b-1.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096: Connection timed out"
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:448:in `fetch_cluster_info'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:402:in `cluster_info'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:102:in `refresh_metadata!'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:56:in `add_target_topics'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/kafka_producer_ext.rb:93:in `initialize'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/kafka_producer_ext.rb:60:in `new'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/kafka_producer_ext.rb:60:in `topic_producer'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/out_kafka2.rb:233:in `write'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin/output.rb:1138:in `try_flush'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin/output.rb:1450:in `flush_thread_run'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin/output.rb:462:in `block (2 levels) in start'
  2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-05-26 15:59:01 +0000 [info]: #0 New topics added to target list: messages
2021-05-26 15:59:01 +0000 [info]: #0 Fetching cluster metadata from kafka://b-1.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096

I am not sure if I missed something or is this not a right configuration.

How do you know fluentd is actually the problem? You're getting a network exception, so double check your VPC firewall and check telnet/netcat connect from the same host where fluentd runs to port 9096 of the brokers — OneCricketeer, May 27 '21 at 12:49

score 0 · Answer 1 · answered Jul 05 '21 at 17:46

I have the same problem and found that it cannot be produced or consumed from the Amazon MSK cluster through a public endpoint because Amazon MSK does not support public endpoints. It is always a private connection.

Possible solutions: There are several methods to connect to your AWS MSK clusters outside of your VPC.

VPN: https://docs.aws.amazon.com/vpc/latest/userguide/vpn-connections.html
VPC Peering: https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html
VPC Transit Gateway: https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html
AWS Direct Connect: https://aws.amazon.com/directconnect/
REST Proxy: A REST proxy can be installed on an instance running within your VPC. REST proxies allow your producers and consumers to communicate to the cluster through HTTP API requests.

Resources: [FAQs]: https://aws.amazon.com/msk/faqs/

Unable to send data from fluentd to Amazon Hosted Kafka

1 Answers1