How to initialize Splunk HTTP Event Collector via Docker Compose and use it with splunk logging driver

Question

I'm trying to set up a local development environment with Docker Compose that bootstraps a Splunk Enterprise server and uses the splunk logging driver on an app server.

Versions:

Docker Engine: 18.06.1-ce
Compose: 1.22.0
Compose File: 3.7
Splunk Enterprise: 7.2.0

My docker-compose.yml file looks like this:

version: "3.7"

services:
  app:
    build: ./app
    command: bash -c "npm run start:docker"
    depends_on:
      - splunk
    environment:
      - NODE_ENV=development
      - SERVER_PORT=8080
    logging:
      driver: splunk
      options:
        splunk-format: "json"
        splunk-insecureskipverify: "true"
        splunk-source: "app"
        splunk-token: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
        splunk-url: "http://splunk:8088"
        tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
    ports:
      - "80:8080"
    volumes:
      - "./app:/usr/src/app"

  splunk:
    environment:
      - SPLUNK_ENABLE_LISTEN=9997
      - SPLUNK_START_ARGS=--accept-license --no-prompt --answer-yes
      - SPLUNK_USERNAME=admin
      - SPLUNK_PASSWORD=password
    hostname: splunk
    image: splunk/splunk:7.2.0
    ports:
      - "8000:8000"
      - "8088:8088"
      - "9997:9997"
    restart: always

In order for this to work as intended, I need to generate an HTTP Event Collector token and make it available to the app service somehow.

I've seen that you can use the environment variable SPLUNK_CMD to run commands, presumably after the Splunk service is up and running, but when I tried using that to generate a token with the CLI, nothing happened. I saw no failure in the logs, and no token under Settings > Data Inputs.

Another issue is that Splunk takes some time to start up, and before it starts listening the app service fails to build because the logging driver cannot connect.

Is it possible to do what I'm trying to do? If so, how?

score 2 · Answer 1 · answered Oct 14 '18 at 05:49

The configuration of the new image (7.2.0) says that you can specify an HTTP Event Collector token with the environment variable https://github.com/splunk/docker-splunk/blob/48d5322bc574792a5bfbfe8f68769aa16e7688b7/documentation/ADVANCED.md#valid-enterprise-environment-variables

But I don't think it works for single instance after looking at https://github.com/splunk/splunk-ansible/search?q=set_as_hec_receiver.yml&unscoped_q=set_as_hec_receiver.yml - seems like that playbook will be executed only for heavy-weight-forwarder and indexer.

Alternatively, if you will look at the "legacy"/community supported image you will find a different way of doing that. As an example, you can take a look at the app-boilerplate that we use at Outcold Solutions for developing Splunk apps https://github.com/outcoldsolutions/splunk-app-boilerplate, where we:

Map configurations https://github.com/outcoldsolutions/splunk-app-boilerplate/blob/master/Makefile#L23
Copy it over https://github.com/outcoldsolutions/splunk-app-boilerplate/blob/master/Makefile#L26

To solve this issue "Another issue is that Splunk takes some time to start up, and before it starts listening the app service fails to build because the logging driver cannot connect." - please take a look on option splunk-verify-connection (see https://docs.docker.com/config/containers/logging/splunk/#splunk-options), in that way it will keep retrying to send the data over and over till the HTTP Event Collector will be available.

As alternative to splunk-verify-connection you can also use a different approach of forwarding logs to Splunk, by using Outcold Solutions collector, that forwards container logs from JSON logs. It is easy to install https://www.outcoldsolutions.com/docs/monitoring-docker/v5/installation/, and you will be able to use an application for monitoring your docker environments as well https://splunkbase.splunk.com/app/3723/

Thank you, this was very helpful -- particularly the link to the advanced configuration docs for the Splunk Docker image, which I had not found in my searches. That set me on a good path to finding a solution which, as you alluded to, may involve using the `splunk_frontend_forwarder`. I looked at your alternative solution, but that appears to be a paid service and I'd really like to get this working just using Splunk and Docker Compose if possible. — Shaun Scovil, Oct 18 '18 at 02:44
Correct, it is a paid application. We do offer free development trial licenses for 6 months with possibility to extend. See https://www.outcoldsolutions.com/contact/ — outcoldman, Oct 18 '18 at 06:25

How to initialize Splunk HTTP Event Collector via Docker Compose and use it with splunk logging driver

1 Answers1