How to group logs messages in logstash using grok?

Question

 [25-Dec-2015 08:06:45] 0:: users to chek for delete
 [25-Dec-2015 08:08:44] 0:: users to chek for delete
 [25-Dec-2015 08:10:44] 3:: users to chek for delete
 [25-Dec-2015 08:10:44] Expected response code 200, got 404

     {
         "error": {
          "errors": [
           {
            "domain": "global",
            "reason": "notFound",
            "message": "Resource Not Found: userKey"
           }
          ],
          "code": 404,
          "message": "Resource Not Found: userKey"
         }
        }

    [06-Nov-2015 19:24:19 GMT] PHP Fatal error:  Class 'Test\Test\Api\Resources\Authenticate1234' not found in /apps/test/src/Test/Test/Api/Resources/ResourceFactory.php on line 10
    [06-Nov-2015 19:24:19 GMT] PHP Stack trace:
    [06-Nov-2015 19:24:19 GMT] PHP   1. {main}() /apps/test/public/api.php:0
    [06-Nov-2015 19:24:19 GMT] PHP   2. Test\Test\Api\ApiController->handleRequest() /apps/test/public/api.php:13
    [06-Nov-2015 19:24:19 GMT] PHP   3. Test\Test\Api\Resources\ResourceFactory->create() /apps/test/src/Test/Test/Api/ApiController.php:14

Above is sample of my log file. I need to filter out each message. the problem is with writing filters. The first 3 lines are three different errors.

[25-Dec-2015 08:06:45] 0:: users to chek for delete
[25-Dec-2015 08:06:45] 0:: users to chek for delete
[25-Dec-2015 08:06:45] 3:: users to chek for delete

The fourth error is a error with JSON message. I need to separate this block from above.

    [25-Dec-2015 08:10:44] Expected response code 200, got 404
     {
         "error": {
          "errors": [
           {
            "domain": "global",
            "reason": "notFound",
            "message": "Resource Not Found: userKey"
           }
          ],
          "code": 404,
          "message": "Resource Not Found: userKey"
         }
        }

The fifth error is PHP stack trace.

        [06-Nov-2015 19:24:19 GMT] PHP Fatal error:  Class 'Test\Test\Api\Resources\Authenticate1234' not found in /apps/test/src/Test/Test/Api/Resources/ResourceFactory.php on line 10
        [06-Nov-2015 19:24:19 GMT] PHP Stack trace:
        [06-Nov-2015 19:24:19 GMT] PHP   1. {main}() /apps/test/public/api.php:0
        [06-Nov-2015 19:24:19 GMT] PHP   2. Test\Test\Api\ApiController->handleRequest() /apps/test/public/api.php:13
        [06-Nov-2015 19:24:19 GMT] PHP   3. Test\Test\Api\Resources\ResourceFactory->create() /apps/test/src/Test/Test/Api/ApiController.php:14

Is it possible to design a grok filter to match these 3 conditions?

score 1 · Answer 1 · edited Dec 28 '15 at 19:17

Use the multiline option. For example:

filter {
    multiline {
        negate    => true
        pattern   => "^\["
        what      => "previous"
    }
}

The result should look like this:

[06-Nov-2015 19:24:19 GMT] PHP Fatal error:  Class 'Test\Test\Api\Resources\Authenticate1234' not found in /apps/test/src/Test/Test/Api/Resources/ResourceFactory.php on line 10
PHP Stack trace:
PHP   1. {main}() /apps/test/public/api.php:0
PHP   2. Test\Test\Api\ApiController->handleRequest() /apps/test/public/api.php:13
PHP   3. Test\Test\Api\Resources\ResourceFactory->create() /apps/test/src/Test/Test/Api/ApiController.php:14

The problem is with date before each line. – Naveedh Dec 29 '15 at 06:40 — Naveedh, Dec 29 '15 at 06:40

score 0 · Answer 2 · answered Dec 28 '15 at 17:53

Your first step is to get the multi-line json error into one logstash event. Check out the multiline codec or filter. Then, I would recommend using one grok{} stanza to pull the datetime off the line, and then use another grok stanza to process the remaining part of the line.

How to group logs messages in logstash using grok?

2 Answers2