Stack Overflow
Stack Overflow

Questions tagged [logstash-grok]

Grok is an abstraction on top of regular expressions to allow easy parsing of unstructured text into structured and queryable form.

Parse arbitrary text and structure it.

Grok is a great way to parse unstructured log data into something structured and queryable.

This tool is perfect for syslog logs, apache and other webserver logs, mysql logs, and in general, any log format that is generally written for humans and not computer consumption.

Logstash ships with about 120 patterns by default. You can find them here: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns. You can add your own trivially. (See the patterns_dir setting)

If you need help building patterns to match your logs, you will find at:

1552 questions
5
votes
2 answers

Trim field value, or remove part of the value

I am trying to adjust path name so that it no longer has the time stamp attached to the end. I am input many different logs so it would be impractical to write a conditional filter for every possible log. If possible I would just like to trim the…
virus.cmd
  • 335
  • 5
  • 13
5
votes
1 answer

Logstash: Keeping a value across events

I have a date that is only present once in every log file and I am trying to add this date to all following events after it has been matched once, making it act like a global variable in some ways. (The date is at the top of the document and I am…
halpsb
  • 1,106
  • 2
  • 18
  • 28
5
votes
1 answer

EVAL inside grok logstash

I am trying to add new filed in grok filter which supposed to an arithmetic expression of the fields that are extracted by grok match command. Unfortunately was not able to figure out the correct syntax for that... Anybody? I found somewhere that…
user3413303
  • 51
  • 1
  • 2
5
votes
1 answer

Regex - discard text after comma character

If I have the text: test: firstString, blah: anotherString, blah:lastString How can I get the text "firstString" My regex is: test:(.*), EDIT Which brings back firstString, blah: anotherString, but I only need to bring back the text…
user86834
  • 5,357
  • 10
  • 34
  • 47
5
votes
5 answers

Regular expression to extract part of a file path using the logstash grok filter

I am new to regular expressions but I think people here may give me valuable inputs. I am using the logstash grok filter in which I can supply only regular expressions. I have a string like this /app/webpf04/sns882A/snsdomain/logs/access.log I want…
flyasfish
  • 119
  • 2
  • 3
  • 11
4
votes
1 answer

Parsing nested Apache Airflow log lines with Logstash

I am wrestling with ingesting Apache Airflow logs into Elasticsearch, using Logstash filters to parse the log lines. One thing that I am struggling with getting my head around how to do appropriately is to handle cases where log lines are nested,…
Martin Secher Skeem
  • 141
  • 1
  • 5
4
votes
1 answer

Logstash with helm in Kubernetes : grok filter not working

I installed a filebeat -> logstash -> elasticsearch -> kibana stack in Kubernetes with helm charts : helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator helm install --name elastic --namespace monitoring…
Michaël L
  • 469
  • 5
  • 17
4
votes
0 answers

How to debug Grok parse failure

I have two somewhat similar logstash configs. One works ok and the other not and I don't know how to proceed. First working config: input { stdin { type => "dec" } } filter { if [type] == "dec" { if "SYN/ACK" in [message] or "ACK" in…
RR1
  • 333
  • 1
  • 6
  • 13
4
votes
1 answer

What is the correct way to have multiple grok patterns in a Logstash filter?

What is the valid logstash config of these 2 options? else if [pipeline] == "tomcat_all" { grok { match => [ "message",…
Dennis
  • 2,866
  • 7
  • 32
  • 49
4
votes
1 answer

logstash grok, parse a line with json filter

I am using ELK(elastic search, kibana, logstash, filebeat) to collect logs. I have a log file with following lines, every line has a json, my target is to using Logstash Grok to take out of key/value pair in the json and forward it to elastic…
user84592
  • 4,750
  • 11
  • 55
  • 91
4
votes
2 answers

grok not reading a word with hyphen

This is my grok pattern 2017-09-25 08:58:17,861 p=14774 u=ec2-user | 14774 1506329897.86160: checking for any_errors_fatal I'm trying to read the user but it's giving only ec2 , it's not giving the full word Sorry i'm newer to the grok filter My…
Vicky
  • 819
  • 2
  • 13
  • 30
4
votes
2 answers

Grok/Oniguruma pattern to match first IP from X-Forwarded-For header

For this issue I'm trying to create a grok pattern, which matches the first IP from the X-Forwarded-For header in a nginx log. A log line typically looks like this: 68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET…
sepal
  • 43
  • 2
  • 5
4
votes
1 answer

How to find time difference in milliseconds from two datetime stamp field using ruby

I need to get time difference from two time stamp fields, i.e., My input : "requestTime" => "2016-12-27 18:35:13:833", "responseTime" => "2016-12-27 18:35:13:834", I Need to get time diff as 1 milliseconds as a result. I used this code : …
Jeeva N
  • 431
  • 1
  • 5
  • 17
4
votes
2 answers

How to combine characters to create custom pattern in GROK

I'm new to logstash and grok and have a question regarding a pattern. Jul 26 09:46:37 The above content contains %{MONTH} %{MONTHDAY} %{TIME} and white spaces. I need to know how to combine all these and create a pattern…
Jerald Sabu M
  • 1,200
  • 3
  • 16
  • 19
4
votes
2 answers

logstash grok Capturing repeating pattern

I have an input that is like the following 2015-12-02T12:39:57.546461+08:00 222.55.201.19 c): start_time="2015-12-02 ...some other log info... src=192.168.1.153 dst=8.8.8.8 src_port=51145 dst_port=13 ...some other log info... …
Shawn Sim
  • 545
  • 1
  • 5
  • 17
1 2 3
…
99 100