I've been working to create a syslog application in which I have to collect different kinds of logs generated using Fluentd and send them to Cassandra for analysis later on. I have to use Kafka as a kind of buffer or a stream processor. For using Kafka with Fluentd, I have to configure Fluentd.
I have installed Fluentd, Cassandra, Kafka and Zookeeper, all of them are running without any errors independently. But when I try to connect Kafka to fluentd using the fluent.conf file, I encounter multiple errors.
My fluent.conf file is :
<source>
@type syslog
bind 0.0.0.0
port 5140
tag default
protocol_type udp
source_hostname_key source_host
<parse>
message_format auto
</parse>
</source>
<match **>
@type stdout
</match>
<match app.**>
@type kafka_buffered
zookeeper localhost:2181
zookeeper_path root/attackfence/kafka_2.11-2.2.0/zookeeeper
partition_key partition
partition_key_key partition_key
message_key_key message_key
default_topic nil
default_partition_key nil
default_message_key nil
output_data_type json
output_include_tag false
output_include_time false
exclude_topic_key false
exclude_partition_key false
get_kafka_client_log true
</match>
The errors which I'm getting are:
2019-05-23 18:22:44 +0530 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `each' for nil:NilClass"
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluent-plugin-kafka-0.9.4/lib/fluent/plugin/out_kafka_buffered.rb:117:in `refresh_client'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluent-plugin-kafka-0.9.4/lib/fluent/plugin/out_kafka_buffered.rb:208:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/compat/call_super_mixin.rb:42:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:165:in `block in start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `each'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `block in lifecycle'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `each'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `lifecycle'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:164:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:274:in `start'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:219:in `run'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:799:in `run_engine'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:549:in `block in run_worker'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:724:in `main_process'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:544:in `run_worker'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/bin/fluentd:8:in `<top (required)>'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/bin/fluentd:23:in `load'
2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/bin/fluentd:23:in `<main>'
