0

I am trying a simple fluentbit / fluentd test with ipv6, but it is not working.

Configuration from fluentbit side:

[SERVICE]
    Flush         5
    Daemon        off
[INPUT]
    Name              cpu
    Tag               fluent_bit
[OUTPUT]
    Name   forward
    Match  *
    Host   fd00:7fff:0:2:9c43:9bff:fe00:bb
    Port   24000

Configuration from fluentd side:

<source>
  type forward
  bind ::
  port 24000
</source>
~
<match fluent_bit>
  type stdout
</match>

I start up fluentd with command: /usr/sbin/td-agent -c test.conf

Then, I start up fluentbit with command: /opt/td-agent-bit/bin/td-agent-bit -c test.conf

The output shows that there is a problem with communication:

Fluent Bit v1.6.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/11/29 01:53:49] [ info] [engine] started (pid=142)
[2020/11/29 01:53:49] [ info] [storage] version=1.0.6, initializing...
[2020/11/29 01:53:49] [ info] [storage] in-memory
[2020/11/29 01:53:49] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/29 01:53:49] [ info] [sp] stream processor started
[2020/11/29 01:53:53] [error] [io] connection #27 failed to: fd00:7fff:0:2:9c43:9bff:fe00:bb:24000
[2020/11/29 01:53:53] [error] [output:forward:forward.0] no upstream connections available
[2020/11/29 01:53:53] [ warn] [engine] failed to flush chunk '142-1606614829.871139401.flb', retry in 8 seconds: task_id=0, input=cpu.0 > output=forward.0
[2020/11/29 01:53:58] [error] [io] connection #28 failed to: fd00:7fff:0:2:9c43:9bff:fe00:bb:24000
[2020/11/29 01:53:58] [error] [output:forward:forward.0] no upstream connections available
[2020/11/29 01:53:58] [ warn] [engine] failed to flush chunk '142-1606614833.871418916.flb', retry in 6 seconds: task_id=1, input=cpu.0 > output=forward.0
[2020/11/29 01:54:01] [error] [io] connection #29 failed to: fd00:7fff:0:2:9c43:9bff:fe00:bb:24000
[2020/11/29 01:54:01] [error] [output:forward:forward.0] no upstream connections available
[2020/11/29 01:54:01] [ warn] [engine] chunk '142-1606614829.871139401.flb' cannot be retried: task_id=0, input=cpu.0 > output=forward.0

However when I run the command without using the config file but passing the parameters, it works:

# /opt/td-agent-bit/bin/td-agent-bit -i cpu -t fluent_bit -o forward://[fd00:7fff:0:2:9c43:9bff:fe00:bb]:24000 -v
Fluent Bit v1.6.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/11/29 01:56:53] [ info] Configuration:
[2020/11/29 01:56:53] [ info]  flush time     | 5.000000 seconds
[2020/11/29 01:56:53] [ info]  grace          | 5 seconds
[2020/11/29 01:56:53] [ info]  daemon         | 0
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info]  inputs:
[2020/11/29 01:56:53] [ info]      cpu
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info]  filters:
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info]  outputs:
[2020/11/29 01:56:53] [ info]      forward.0
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info]  collectors:
[2020/11/29 01:56:53] [ info] [engine] started (pid=151)
[2020/11/29 01:56:53] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/11/29 01:56:53] [debug] [storage] [cio stream] new stream registered: cpu.0
[2020/11/29 01:56:53] [ info] [storage] version=1.0.6, initializing...
[2020/11/29 01:56:53] [ info] [storage] in-memory
[2020/11/29 01:56:53] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/29 01:56:53] [debug] [forward:forward.0] created event channels: read=20 write=21
[2020/11/29 01:56:53] [debug] [router] default match rule cpu.0:forward.0
[2020/11/29 01:56:53] [ info] [sp] stream processor started
[2020/11/29 01:56:57] [debug] [task] created task=0x7f9b4e8580a0 id=0 OK
[2020/11/29 01:56:57] [debug] [output:forward:forward.0] request 5525 bytes to flush
[2020/11/29 01:56:57] [debug] [upstream] KA connection #27 to fd00:7fff:0:2:9c43:9bff:fe00:bb:24000 is now available
[2020/11/29 01:56:57] [debug] [task] destroy task=0x7f9b4e8580a0 (task_id=0)
[2020/11/29 01:57:02] [debug] [task] created task=0x7f9b4e8580a0 id=0 OK
[2020/11/29 01:57:02] [debug] [output:forward:forward.0] request 4420 bytes to flush

Does anyone understand what the difference is and how i can rectify this problem?

kiwo
  • 128
  • 1
  • 8
  • It appears that you have two processes trying to use the same port, but that does not work. When you start without the configuration file, it will choose an ephemeral port. Only the server side should be configured with a port number, the client side should choose its own. – Ron Maupin Nov 29 '20 at 06:05
  • Hi Ron. Sorry, i didn't mention that my server and clients are on different containers. – kiwo Nov 30 '20 at 13:31

1 Answers1

0

I resolved this issue by taking the source from the latest fluent-bit project and compiling it. Then I replaced /usr/sbin/td-agent-bit with the newly built fluent-bit (as td-agent).

kiwo
  • 128
  • 1
  • 8