1

I have some questions, how I can set telegraf.conf file for collect logs from the "zimbra.conf" file? Now I tried to use this config text, but it does not work :((( I want to send this logs to grafana

One of the lines "zimbra.conf" for example:

Oct 1 10:20:46 webmail postfix/smtp[7677]: BD5BAE9999: to=user@mail.com, relay=mo94.cloud.mail.com[92.97.907.14]:25, delay=0.73, delays=0.09/0.01/0.58/0.19, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 4C25fk2pjFz32N5)

And I do not understand exactly how works the "grok_patterns ="

[[inputs.tail]]
  files = ["/var/log/zimbra.log"]
  from_beginning = false
  grok_patterns = ['%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST} %{DATA:program}(?:\[%{POSINT}\])?: %{GREEDYDATA:message}']
  name_override = "zimbra_access_log"
  grok_custom_pattern_files = []
  grok_custom_patterns = '''
  TS_UNIX %{MONTH}%{SPACE}%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}
  TS_CUSTOM %{MONTH}%{SPACE}%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}
  '''
  grok_timezone = "Local"
  data_format = "grok"
Oleg
  • 161
  • 1
  • 2
  • 10
  • The question is a bit vague, are you trying to use telegraph to send zimbra data to Graphana? Clarify your question a bit. Otherwise, great question. – Andrew Carr Oct 08 '20 at 18:06
  • Yes, correct. I have server with Zimbra, and I have log file /var/log/zimbra.log, so I want to send this log to InfluxDB then using Grafana check this log and create some Query to find some date. – Oleg Oct 13 '20 at 09:18

1 Answers1

0

I have copied your example line into a log file called Prueba.txt wich contains the following lines:

Oct 3 00:52:32 webmail postfix/smtp[7677]: BD5BAE9999: to=user@mail.com, relay=mo94.cloud.mail.com[92.97.907.14]:25, delay=0.73, delays=0.09/0.01/0.58/0.19, dsn=2.0.0, status=sent (250 2.0$
Oct 13 06:25:01 webmail systemd-logind[949]: New session 229478 of user zimbra.
Oct 13 06:25:02 webmail zmconfigd[27437]: Shutting down. Received signal 15
Oct 13 06:25:02 webmail systemd-logind[949]: Removed session c296.
Oct 13 06:25:03 webmail sshd[28005]: Failed password for invalid user julianne from 120.131.2.210 port 10570 ssh2

I have been able to parse the data with this configuration of the tail.input plugin:

[[inputs.tail]]
  files = ["Prueba.txt"]
  from_beginning = true
  data_format = "grok"
  grok_patterns = ['%{TIMESTAMP_ZIMBRA} %{GREEDYDATA:source} %{DATA:program}(?:\[%{POSINT}\])?: %{GREEDYDATA:message}']


  grok_custom_patterns = '''
    TIMESTAMP_ZIMBRA (\w{3} \d{1,2} \d{2}:\d{2}:\d{2})
  '''

  name_override = "log_frames"

You need to match the input string with regular expressions. For that there are some predefined patters such as GREEDYDATA = .* that you can use to match your input (another example will be NUMBER = (?:%{BASE10NUM}) BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))) . You can also define your own patterns in grok_custom_patterns. Take a look at this website with some patters: https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Apx-GrokPatterns/GrokPatterns_title.html

In this case I defined a TIMESTAMP_ZIMBRA pattern for matching Oct 3 00:52:32 and Oct 03 00:52:33 alike inputs.

Here is the collected metric by Prometheus:

# HELP log_frames_delay Telegraf collected metric
# TYPE log_frames_delay untyped
log_frames_delay{delays="0.09/0.01/0.58/0.19",dsn="2.0.0",host="localhost.localdomain",message="BD5BAE9999:",path="Prueba.txt",program="postfix/smtp",relay="mo94.cloud.mail.com[92.97.907.14]:25",source="webmail",status="sent (250 2.0.0 Ok: queued as 4C25fk2pjFz32N5)",to="user@mail.com"} 0.73

P.D.: Ensure that telegraf has access to the log files.

kevin
  • 988
  • 12
  • 23
  • Hello and thank you very much for your help. Now I am trying to use your example. How i should ensure that telegraf has access to the log files? – Oleg Oct 13 '20 at 08:40
  • I did not understand how put text with `code` parameters, so I copied my tail config info txt file :-) https://yadi.sk/d/mwN2RqoikX5ctg I tried to collect data from Grafana and from Chronograf, but can not :( – Oleg Oct 13 '20 at 11:02
  • I think that first, you should ensure that telegraf is exporting the metrics. For that you could try to run it in debug mode with telegraf --config ./telegraf.config --debug. Thus, you would be able to see if it is exporting metrics and you could also check if it has permisson to read the file or not (by adding privileges to the telegraf user). After that, ensure that those metrics are properly inserted in InfluxDB (maybe you have to modify something in the [outputs] of telegraf configuration), and finally, try to visualize them in grafana by adding the InfluxDB data source. – kevin Oct 13 '20 at 11:25
  • I think my telegraf services is working, because I have metrics in Grafana from this server where located Zimbra This is screenshot with metrics from server with zimbra https://ibb.co/ZKk8748 And this is a logs of debug telegraf https://yadi.sk/d/Th738DUrRGNISg – Oleg Oct 13 '20 at 11:58
  • Take a look at the output of the debug: D! Grok no match found for: "Oct 13 06:25:01 webmail systemd-logind[949]: New session 229478 of user zimbra." The patter is not matching the input. I will try to make the patter a bit more generic and update it. – kevin Oct 13 '20 at 12:28
  • Ok, I have updated the pattern and the input examples. For those 5 lines, that pattern is working for me. It matches every message in the form: Zimbra_date source program[number]: message – kevin Oct 13 '20 at 12:36
  • Cool! What should I change exactly, could you please tell me? – Oleg Oct 13 '20 at 13:46
  • Just the grok_patterns line. I have edited it in the answer. – kevin Oct 13 '20 at 14:03
  • Ok, I edit grok_patterns line and now I do not have errores. This is awesome! But when I try to get this logs in Grafana, i can find how i can view. Add panel, but I did not have lines with logs when i do some query https://yadi.sk/d/x0dzkTR7G3DXpQ – Oleg Oct 13 '20 at 15:13
  • You don't have numeric logs, so you could use Table Panel https://community.grafana.com/t/can-show-string-values-on-panels/16000 to show the captured metrics. – kevin Oct 13 '20 at 15:45
  • Hello! :) Thank you for your answers! I can find Table Panel in Grafana or I should import this dashboard? – Oleg Oct 14 '20 at 07:18
  • On Panel -> Visualization you have the Table option. For the queries, with Prometheus data source you can choose to visualize the data in Time Series format or in Table format, however, I do not know how this is managed with InfluxDB data sources. – kevin Oct 14 '20 at 07:37