1

I need to use my log's timestamp as @timestamp. I've tried different methods from the internet but none of the worked in my case

input {
    s3 {
        bucket => "*"
        access_key_id => "*"
        secret_access_key => "*"
        prefix => "*"
        backup_to_bucket => "*"
        backup_add_prefix => "*"
        region => "*"
        delete => *
    }
}

filter {
    grok {
        match => { "message" => "%{IPORHOST:clientip} \[%{TIMESTAMP_ISO8601:logtimestamp}\] %{WORD:protocol} %{GREEDYDATA:uri_path} \"%{WORD:verb} %{DATA:partial_request} HTTP/%{NUMBER:httpversion}\" \"%{GREEDYDATA:User_Agent}\" \"%{GREEDYDATA:request}\" %{NUMBER:response} %{NUMBER:ret1} %{NUMBER:ret2} "}
    }



date {
    match => ["logtimestamp", "ISO8601"]
    target => "@timestamp"
  }

mutate
    {
         remove_field => [ "message" ]
    }
}
output {
    elasticsearch {
        hosts => ["endpoint:9200"]
        index => "mywebsite.com"
    }
    stdout {
        codec => rubydebug
    }

}

sample log entry:

46.229.168.134 [17/Jun/2019:08:00:19 +0000] https www.mywebsite.com "GET /somefolder/somefolder/request HTTP/1.1" "Mozilla/5.0 (compatible; someBot/3~bl; +http://www.somebot.com/bot.html)" "" 200 48260 8

it doesn't show any errors but @timestamp is different from the log's timestamp even though I tried converting it to my browser's timezone which is how the kibana is configured.

Artisan Ilustrado
  • 25
  • 3
  • I just tried your configuration with the sample log message and got a "_grokparsefailure" tag in the resulting document. Are you sure you're not getting the same and your grok patter is correct? – mihomir Jun 20 '19 at 09:35
  • I believe your date is not in TIMESTAMP_ISO8601 format: https://github.com/hpcugent/logstash-patterns/blob/master/files/grok-patterns You can see that the TIMESTAMP_ISO8601 has "-" as separators and not "/". Try to replace the TIMESTAMP_ISO8601 part of your grok pattern with a DATA and then update the "date" filter with the correct pattern – mihomir Jun 20 '19 at 10:03
  • 1
    @mihomir I tried what you told me and now it's not uploading to elasticsearch . – Artisan Ilustrado Jun 24 '19 at 05:28
  • Does it still print anything in the console from the stdout output? What about the "_grokparsefailure"? Wasn't that tag there before? Is it there now? – mihomir Jun 25 '19 at 12:32

0 Answers0