0

How can we push Python Application logs to Splunk. what are the prerequisite.Should we first convert out logs in structured format(key-value based) before sending to Splunk?

There is a package splunk_handler for pushing logs to splunk but i could not found any documentation with practical working example.I just have gitbub page for this package but that does not have any practical implementation.

Please help on this

Sarvendra Singh
  • 109
  • 1
  • 1
  • 9

1 Answers1

2

There are a couple of ways to do that. The first is to install Splunk's Universal Forwarder (UF) and have it monitor the file(s) where the logs are written. The UF will handle sending the logs to Splunk. You do not have to convert the logs, but may have to configure Splunk to interpret them correctly.

The second method is to send the logs directly to Splunk's HTTP Event Collector (HEC). There is a Python class to do that available on GitHub at https://github.com/georgestarcher/Splunk-Class-httpevent .

RichG
  • 9,063
  • 2
  • 18
  • 29
  • Thanks @RichG ...In your opinion which id better way out of the two... and UF needs to be installed at splunk Server side or at python application side – Sarvendra Singh Nov 15 '21 at 15:18
  • 1
    I prefer to use the UF because it doesn't require application changes. It does, however, require the log files be on a platform supported by Splunk. If that's not the case then use the HEC method. The UF is installed where the log files are. – RichG Nov 15 '21 at 15:27
  • @RichG - if you can run Python on it... the platform is *probably* supported to run the UF :) – warren Nov 16 '21 at 15:21
  • Thanks @RichG I checked in my organisation I have to follow HEC method. apart from github link can you please suggest any other documentation/links wih practical step by step implementation..i could not get much from github .Also to move logs to splunk is there any prerequisite ..like our logs should be completely structured(key value dictionary based) or we can move String formatted logs also to splunk using HEC. thanks in advance. – Sarvendra Singh Nov 17 '21 at 15:57
  • HEC accepts either raw or JSON-formatted logs. See https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/FormateventsforHTTPEventCollector and https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/HECExamples . – RichG Nov 18 '21 at 12:25
  • @RichG Thanks..So we need fire the curl HEC requests from our python application after each logger statement (there are multiple self.logger.info statements in application) or we need to execute these curl requests once the python job is completed and we can send entire log file created in one go with single curl command.. – Sarvendra Singh Nov 19 '21 at 09:09
  • To be clear, you do not have to execute curl commands from within your python jobs. Use python-native methods to do the same thing curl does in the examples in the documentation. – RichG Nov 19 '21 at 12:41