Trying to log to Splunk using logback appender

Question

I'm trying to log to splunk directly in my Java application using the logback appender for splunk.

Nothing seems to be going to splunk, but when I manually issue a post command in a REST client i'm seeing my data in splunk.

I wasn't able to get the official splunk logback test to work either.

logback.xml

  <appender name="SPLUNK" class="com.splunk.logging.HttpEventCollectorLogbackAppender">
    <url>http://mySplunkUrl:8088/services/collector</url>
    <token>1234566789</token>
    <disableCertificateValidation>true</disableCertificateValidation>
    <batch_size_count>1</batch_size_count>
    <layout class="ch.qos.logback.classic.PatternLayout">
      <pattern>%msg</pattern>
    </layout>
  </appender>

  <root level="trace">
    <appender-ref ref="SPLUNK" />
  </root>

</configuration>

Unit test

@Test
public void splunkLogger() {
    Logger logger = LoggerFactory.getLogger(LogFactoryTest.class);
    Date date = new Date();
    String jsonMsg = String.format("{event:'CancerCenterTest'}");
    logger.info("CancerCenterTest");
    logger.info(jsonMsg);
}

This is the documentation I was trying to use: http://dev.splunk.com/view/splunk-logging-java/SP-CAAAE7M

Am I missing something obvious?

EDIT Here's a link to my project - https://github.com/toymachiner62/splunk-logging

Can you send a .zip of your project to devinfo@splunk.com? I can help out if I have a bit more context on how your project is configured — Shakeel, Nov 23 '16 at 18:30
@Catfish can you post the import lines for the unit test. It is not clear if the test is using Slf4j or Logback `Logger` and `LoggerFactory` classes. What logging dependencies are being used? Is this a maven project? If so, can you paste the relevant part of the `pom.xml`. Also it would be nice if you could share the zip with other readers as well ;) — anttix, Nov 28 '16 at 20:07
@anttix ok I updated the question with a link and put my code on github. — Catfish, Nov 28 '16 at 20:18

anttix · Accepted Answer · 2016-11-29T17:02:29.860

9

The problem seems to be that a unit test JVM will exit before Splunk background thread completes the log upload.

Logback adapter for Splunk uses a background thread for sending messages to the server in batches. Batching is controlled by batch_interval, batch_size_bytes and batch_size_count parameters. Even if they are all set to very low levels, the unit test JVM will likely exit and kill the thread before it completes.

Try adding a sleep to the end of the test method e.g.

Thread.sleep(5000);

Also the send may be failing due to some error. These can be surfaced by adding this snippet of code to catch Splunk sender failures:

HttpEventCollectorErrorHandler.onError(new HttpEventCollectorErrorHandler.ErrorCallback() {
    public void error(final List<HttpEventCollectorEventInfo> events, final Exception ex) {
        // FIXME: Dumping stack trace to STDERR is not suitable for production use !
        ex.printStackTrace();
    }
});

Please note that Splunk HttpEventCollectorSender does not seem to be setting any HttpClient timeouts so if the endpoint is unreachable it could take minutes for the connection to time out and an error message to appear.

EDIT: As it turns out there was also an issue with the url parameter in logback.xml . The URL must be specified without a path e.g. http://mySplunkUrl:8088/ not http://mySplunkUrl:8088/services/collector

edited Nov 29 '16 at 17:02

answered Nov 28 '16 at 21:25

anttix

7,709
1
24
25

Adding `Thread.sleep(5000)` didn't work. I Also added this project as a dependency to another project and ran it on my server and it still didn't send logs to splunk. – Catfish Nov 29 '16 at 16:23
Ok so i figured out that my logback.xml was being overwritten by a dependency's logback.xml when i'm running my project on my server. I added that snippet to see splunk sender failures and I'm now getting this error `{"text":"The requested URL was not found on this server.","code":404}`. I've directly posted to the url that I have in my appender though.. – Catfish Nov 29 '16 at 16:43
1

Looking at the `HttpEventCollectorSender ` code it seems that a path (`/services/collector/event/1.0`) gets appended to the URL automatically. I would try to strip the tail end of the URL in `logback.xml` e.g. `http://example.com:8088/services/collector` -> `http://example.com:8088/` – anttix Nov 29 '16 at 16:52
Bingo! That worked! Can you point me to the documentation where you found that the url is appended? – Catfish Nov 29 '16 at 16:54
I used the most auhtoritative documentation available :) The source code of [HttpEventCollectorSender line 108](https://github.com/splunk/splunk-library-javalogging/blob/master/src/main/java/com/splunk/logging/HttpEventCollectorSender.java#L108) – anttix Nov 29 '16 at 16:56
1

Now to be honest, the Splunk documentation linked to from the question does specify URL in a format `%scheme%://%host%:%port%` however since it's represented in a placeholder format, it's hard to read and thus easy to overlook. – anttix Nov 29 '16 at 17:05
Thanks for pointing out that the URL must be specified without a path e.g. http://mySplunkUrl:8088/ not http://mySplunkUrl:8088/services/collector – Adi Feb 05 '18 at 07:16

Trying to log to Splunk using logback appender

1 Answers1