0

I'm new to the log-stash and grok, I need to parse very custom log files. I can't find anywhere a good tutorial to get this done. Tried the syslog example but it's not working in my case.

Example:

Nov 19 00:06:37 srv-fe-05 ssh-server-g3: 2037 Sft_server_open_dir, "2037 Sft_server_open_dir, Directory name: /home/folder1/input, File handle: 007800B000782170, "Success", Session-Id: 162351"

The items I'm looking to extract:

  1. Timestamp: Nov 19 00:06:37 will be added to the current year and stored in the elastic search as timestamp.
  2. Server host srv-fe-05
  3. Folder name folder1 from /home/folder1/input
  4. Success status "Success"
  5. Session-Id: 162351 from Session-Id: 162351

Any help or directions would be appreciated.

Following the answer I came up with this pattern:

%{SYSLOGTIMESTAMP:logTimestamp} %{USERNAME:sftpServer} %{USERNAME:processName}: %{INT:operationType} %{WORD}, \"%{INT} %{WORD}, %{WORD} %{WORD}: /%{WORD}/%{WORD:clientName}/%{WORD}, %{WORD} %{WORD}: %{WORD:submissionId}, \"%{WORD:status}\", %{WORD}-%{WORD}: %{INT:sessionId}

My 2 new questions are:

  1. How efficient is it? I mean what are the ways to make it more efficient?
Dmitry R
  • 2,956
  • 4
  • 25
  • 45

1 Answers1

2

First, become familiar with the basic built-in grok patterns. Knowing the basic tools of %{WORD}, %{NUMBER}, etc, will go a long way.

Second, become familiar with the online debuggers (heroku and appspot). They each have advantages over the other.

Third, start slowly. In your example, can you match the date and time from the beginning of the string in the debugger? If so, then try to add a match for the hostname. Keep moving slowly form left to right until you're all done.

If you look in the core patterns, you'll notice that "nov" only appears once, as part of the pattern %{MONTH}. Makes sense, right? But also note that %{MONTH} is part of larger constructs like %{SYSLOGTIMESTAMP} which would match your datetime on one step.

Finally, there are some online guides and tutorials. Here's one.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • Thanks for guidance, is there any tutorials regarding defining custom patterns? As example I got here with my log line: %{SYSLOGTIMESTAMP:logTimestamp}%{SPACE}%{USERNAME:sftpServer}%{SPACE}%{USERNAME:sftpProcessName}%{GREEDYDATA:message} but can't find a way to continue after the ":" char since it's not in the common patterns. How can I define my own patterns and use them in the grok matcher? – Dmitry R Dec 02 '15 at 17:37
  • You can put literals in your patterns. For a single space, I prefer to use " " rather than %{SPACE}. You can also put a colon in your pattern, e.g. %{WORD}:%{WORD}. – Alain Collins Dec 02 '15 at 18:02