1

I was trying to parse a log file with multiple log messages separated per line using Logstash but it failed to do.

My logstash configuration file is as follow:

 file{
                path=> "/home/vm01/Documents/csb-demo/csb-result.txt"
                type=>"csb-alert"
                start_position => "beginning"
                sincedb_path=>"/home/vm01/Documents/sincedb_redefined2"
                codec => multiline
                {
                        pattern => '^\}'
                        negate => true
                        what => previous
                }
        }

with the log file looks like this:

OutOfOrderEventAlert{name='OUT_OF_ORDER_EVENT', description='An event reached the system way too late!', datetime='2018-01-01T05:13:37.150', attachedEvent=CsbEvent{serviceNameReporter='UserService', eventTimestamp=2018-01-01T05:13:37.150, receivedTimestamp=2018-05-10T19:52:19.498, logLevel='INFO', eventName='LOAD_GROUPS_FOR_USER', eventResult='succeeded', eventDescription='User groups were loaded', userId=0, userUsername='Prof. Etta Kallert MBA.', userPasswordInvoked=false, userGroups=null, fileAccessPolicy='', filePathLocal=, filePathRemote=, fileChunks=null, fileChunksSignedUrl=[], deviceId=0, deviceName='', deviceType='', deviceOs='', deviceOsVersion='', deviceTokenRevoked=false, networkAddressReporter=/0.0.0.0:8080, networkAddressSource=/172.18.0.4:46240, networkAddressDestination=/172.18.0.6:8080, networkRequestLength=-1, networkRequestUseragent='', networkResponseLength=2, networkResponseStatusCode=200, networkResponseStatusMessage='OK', networkSessionValidity=true}}
OutOfOrderEventAlert{name='OUT_OF_ORDER_EVENT', description='An event reached the system way too late!', datetime='2018-01-01T06:04:29.550', attachedEvent=CsbEvent{serviceNameReporter='DeviceService', eventTimestamp=2018-01-01T06:04:29.550, receivedTimestamp=2018-05-10T19:52:36.884, logLevel='INFO', eventName='LOAD_DEVICE', eventResult='succeeded', eventDescription='Device information was loaded', userId=0, userUsername='Vadim Mosemann B.A.', userPasswordInvoked=false, userGroups=null, fileAccessPolicy='', filePathLocal=, filePathRemote=, fileChunks=null, fileChunksSignedUrl=[], deviceId=1, deviceName='100dbf1f0940', deviceType='Personal Computer', deviceOs='linuxamd64', deviceOsVersion='4.13.0-26-generic', deviceTokenRevoked=false, networkAddressReporter=/0.0.0.0:8080, networkAddressSource=/172.18.0.4:46240, networkAddressDestination=/172.18.0.6:8080, networkRequestLength=-1, networkRequestUseragent='', networkResponseLength=2597, networkResponseStatusCode=200, networkResponseStatusMessage='OK', networkSessionValidity=true}}
IllegalCspFileEventAlert{ name='Illegal CSP file event', eventname='REST.GET.OBJECT', description='Invalid URL, parameters are missing.', username='null', datetime='2018-01-01T09:49:45', objectkey='rYhWZk22OF4eCdkeLiy5EXtT5SicVoIpUtzvFEw7TXTanbP7B620hNwDLe9B', url='/rYhWZk22OF4eCdkeLiy5EXtT5SicVoIpUtzvFEw7TXTanbP7B620hNwDLe9B?X-Amz-Date=20180101T095013Z&X-Amz-SignedHeaders=host&X-Amz-Expires=29&X-Amz-Credential=AKIAJJVXZ45BX4HD4THQ%2F20180131%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&X-Amz-Algorithm=AWS4-HMAC-SHA256'}
IllegalCspFileEventAlert{ name='Illegal CSP file event', eventname='REST.PUT.OBJECT', description='A user permormed a cloud storage operation without a corresponding csb operation.', username='Dipl.-Ing. Florian Dowerg B.Sc.', datetime='2018-01-01T06:19:41', objectkey='497wfGnEGOCk7TonSp9wD9pWMxWZOpvEQFMavs6t55bM5OpjxyKsAjWrnnTS', url='/497wfGnEGOCk7TonSp9wD9pWMxWZOpvEQFMavs6t55bM5OpjxyKsAjWrnnTS?X-Amz-Date=20180101T062009Z&USERNAME=Dipl.-Ing.%20Florian%20Dowerg%20B.Sc.&X-Amz-Expires=29&X-Amz-SignedHeaders=host&X-Amz-Credential=AKIAJJVXZ45BX4HD4THQ%2F20180131%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&X-Amz-Algorithm=AWS4-HMAC-SHA256'}

Is there something wrong with the my Logstash configuration?

Ihsan Haikal
  • 1,085
  • 4
  • 16
  • 42
  • With your configuration, all lines _not_ starting with `}` will be joined with the previous line. Are you sure that's what you want? – baudsp May 15 '18 at 13:18
  • @baudsp That's not what I want. What I want is to separate each line as it ends with } but I am not sure how – Ihsan Haikal May 15 '18 at 13:29
  • You have a) more than one message on one line and you want to split it or you have b) a message spread over multiple lines that you want to regroup? I prefer to clear this before saying something wrong for your use case. – baudsp May 15 '18 at 13:45
  • @baudsp hmm basically one line consists of one message and I would like to separate each line into its own message as the way file input plugin works is that it will just concatenate it into one single document. I have read that multiline works in this case but I am not sure how – Ihsan Haikal May 15 '18 at 13:50
  • Usually the file input put one line in one message, unless you use the multiline codec, which merge multiple lines in one message. – baudsp May 15 '18 at 14:00
  • @baudsp could you suggest me what is the best configuration file for this case? I have tried to use it with and without multiline codec but it still gave me the message with all lines combined into one message – Ihsan Haikal May 15 '18 at 15:12
  • Perhaps it's a line ending issue? By default the line ending used is `\n` in the file input, perhaps try to change the [delimiter](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-delimiter) param to `\r\n` – baudsp May 15 '18 at 15:54
  • @baudsp still does not work with what you suggested just now – Ihsan Haikal May 16 '18 at 09:26
  • I don't have any other idea with the information you've provided. – baudsp May 16 '18 at 10:01

0 Answers0