-1

Actually we are planing to use fluentd on nodes server to fetch logs and forward those logs to our log collector(main Fluentd Server)and then we can transfer the logs to s3 buckets now we are able to write the logs to my s3 bucket but i am getting the error when restart the td-agent in debug mode.

fluent nodes configuration:-

root@ip-172-31-32-115:/var/log/td-agent# cat /etc/td-agent/td-agent.conf
#Apache Access logs
<source>
  @type tail
  format apache2
  path /var/log/apache2/access.log
  pos_file /var/log/td-agent/apache2.access.pos
  tag apache.access
</source>
#Log forwardin
<match apache.access>
  @type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 1s
  phi_threshold 16
  hard_timeout 60s
#primary host

  <server>
    host xx.xx.xx.xx
    port 24224
  </server>
  flush_interval 60s
</match>

log forwarder logs

root@ip-172-31-34-201:/home/ubuntu# td-agent -v 
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:457:read_config: reading config file path="/etc/td-agent/td-agent.conf"
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:323:supervise: starting fluentd-0.12.20
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-mongo' version '0.7.12'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-s3' version '0.6.5'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td' version '0.10.28'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluentd' version '0.12.20'
2016-06-06 18:21:44 +0000 [info]: fluent/agent.rb:125:add_match: adding match pattern="apache2.." type="forward"
2016-06-06 18:21:44 +0000 [info]: plugin/out_forward.rb:141:block in configure: adding forwarding server '172.31.33.196:24224' host="172.31.33.196" port=24224 weight=60 plugin_id="object:11e9668"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail" plugin/out_forward.rb:232:rebuild_weight_array: rebuilding weight array lost_weight=0
2016-06-06 18:21:44 +0000 [info]: plugin/in_tail.rb:499:initialize: following tail of /var/log/apache2/access.log

log Aggregator config:-

#Input
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>
# S3 config
<match s3.*.*>
    @type s3

    aws_key_id AKIERPE6JxxxKXSG3GWSA
    aws_sec_key 31KnCOJdQrxxxrXEw0dYY5RqV9zegTedfrere25F1Hsvw
    s3_bucket rana.sahu
    s3_region us-west-2
    path logs/
    buffer_type           file
    buffer_path           /var/log/td-agent/s3/buffer/td
    buffer_chunk_limit    2m
    buffer_queue_limit    4098
    retry_wait            1s
    max_retry_wait        5s
    disable_retry_limit   true


    time_slice_format %Y%m%d%H
    time_slice_wait 10m
    utc

</match>

it stuck on this thing

When i started the td-agent -v i am getting this error 2016-06-07 07:11:00 +0000 [info]: plugin/in_forward.rb:81:listen: listening fluent socket on 0.0.0.0:24224 2016-06-07 07:11:00 +0000 [error]: fluent/engine.rb:196:rescue in run: unexpected error error_class=Errno::EADDRINUSE error=# 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:innew' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:82:innew' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:82:in listen' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:51:instart' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:111:in block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:110:ineach' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:110:in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:225:instart' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:175:in run' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:597:inrun_engine' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:148:in block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:incall' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:in `main_process'

Please let me know if you have any solution for this.

NIKHIL SHARMA
  • 1
  • 4
  • you have nothing listening on `24224` ? - pay attention to your formatting in your question – Frederic Henri Jun 06 '16 at 20:28
  • The problem isn't in the config. There is probably another td-agent service (or another) already running in that `24224` port. – dieend Jun 07 '16 at 04:51

1 Answers1

0

i got the answer and with these configuration we can forward apache2 logs to log collector and store them into s3 bucket.

Thanks all.

NIKHIL SHARMA
  • 1
  • 4