3

I'm looking for a Docker image of Fluentd with fluent-plugin-splunk-hec plugin to send data to Splunk.

I found that some time ago Fluentd provided such an image, but it's discontinued for some/no reason. Fluentd official documentation points at the outdated repository as well.

Finally, I found something promising at Docker Hub splunk/fluentd-hec which states:

By splunk • Updated 24 minutes ago This image contains fluentd along with the fluent-plugin-splunk-hec.

When I tried to use this image to send data to Splunk like this:

<match **>
   @type splunk_hec
   hec_host XX.XX.XX.XX
   hec_port 8088
   hec_token 00000000-0000-0000-0000-000000000000
</match>

I received an error message:

2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-concat' version '2.4.0'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-jq' version '0.5.1'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.4'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-systemd' version '1.0.2'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluentd' version '1.14.6'
2022-08-09 10:12:55 +0000 [error]: fluent/log.rb:372:error: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'splunk_hec'. Run 'gem search -rd fluent-plugin' to find plugins"

When I checked the image itself I found that there's no fluent-plugin-splunk-hec plugin available inside.

There's yet another option available - build own image using this repo, but maybe I missed something important.

EDIT

I've created a new image using https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.15/debian-stackdriver/Dockerfile, but added plugin to the Gemfile as a single line gem "fluent-plugin-splunk-hec". It works as expected:

2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluentd' version '1.15.1'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-concat' version '2.5.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-google-cloud' version '0.4.10'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.13.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-prometheus' version '2.0.3'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-splunk-hec' version '1.2.13'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-systemd' version '1.0.5'
2022-08-11 12:34:22 +0000 [debug]: fluent/log.rb:309:debug: No fluent logger for internal event
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: using configuration file: <ROOT>

In compare to the example above you can clearly see that fluent-plugin-splunk-hec plugin was loaded.

/EDIT

What's wrong with the image from Splunk or with my config? What's the easiest way to make it working?

Serhii Rohoza
  • 4,287
  • 2
  • 16
  • 29
  • You forgot to ask an actual question. – Turing85 Aug 08 '22 at 19:25
  • While Stack Overflow does permit certain questions about Docker, we require that they (like all questions asked here) be specifically related to programming. This question does not appear to be specifically related to programming, but troubleshooting a particular image, which makes it off-topic here. – Turing85 Aug 08 '22 at 19:36
  • 1
    I would most probably try to contact the developers through a [gitub issue](https://github.com/splunk/fluent-plugin-splunk-hec/issues). – Turing85 Aug 08 '22 at 19:40
  • It's a good idea, but TBH I doubt that the Docker image and the GitHub repo are supported by the same Development team. – Serhii Rohoza Aug 08 '22 at 19:44
  • Here you can find an issue report at GitHub https://github.com/splunk/fluent-plugin-splunk-hec/issues/252 – Serhii Rohoza Aug 14 '22 at 09:13

1 Answers1

0

As it turned out the Docker image splunk/fluentd-hec:1.2.13 uses bundle to start fluentd. So, instead of a command:

fluentd -c /fluentd/etc/fluent.conf  -v

you should use bundle:

bundle exec fluentd -c /fluentd/etc/fluent.conf

As a result, everything works as expected:

2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluentd' version '1.14.6'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-concat' version '2.4.0'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-jq' version '0.5.1'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.4'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-splunk-hec' version '1.2.13'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-systemd' version '1.0.2'
2022-08-16 13:03:28 +0000 [debug]: fluent/log.rb:309:debug: No fluent logger for internal event
Serhii Rohoza
  • 4,287
  • 2
  • 16
  • 29