0

https://stackoverflow.com/a/51825609/16120054

Hi All,

Based on the above link solution, has this to be implemented with pipeline.workers 1 in conf settings?

Can anyone please advise?

Number13
  • 3
  • 4

3 Answers3

0

The aggregate filter needs pipeline.workers set to 1 to work correctly and while the elapsed filter does not says anything about the number of workers in the documentation, it is also recommended to set the pipeline.workers to 1.

Both of these filters needs the events to pass through the pipeline on the same thread, so to do that you need pipeline.workers set to 1.

leandrojmp
  • 7,082
  • 2
  • 19
  • 24
  • Then I am a in a bit of fix. I have one conf file which is supposed to read 4 different directories with about 20 logs in each. With one worker, the elapsed is joining start_tag from dir1-file20 to dir2-file1, which is causing false data to be parsed into ES. I have already selected sort direction and sort order in LS to make sure that data is read in order. But still this is happening. – Number13 Jun 10 '21 at 13:09
  • How can I make sure that the log read form that dir1-file1 is done and dir2-file1 isn't joined from previous start and end tags by elapsed. This would solve a very big issue for me if you can help. – Number13 Jun 10 '21 at 13:12
  • You need to share your full pipeline to show what you are doing, but you can use conditionals to select when a filter should be applied or not. – leandrojmp Jun 10 '21 at 13:46
  • Please, do not answer your own question to add new information is this would make things confusing, edit your original question to add the new information. Also do not tag people, just wait. – leandrojmp Jun 11 '21 at 13:11
  • Noted! Will edit the question for clarity. My concern is still the same as my earlier comment of elapsed filter joining "elapsed_time" from other dir's files. For that reason, I was thinking of other ways to cope with the elapsed_time behavior to be ordered and hence the answer. – Number13 Jun 11 '21 at 14:20
  • The elapsed filter needs a unique id, do you have this unique id in your logs? Currently you are using the `host` field as the unique id, if you have multiple dirs in this host, the logs will be mixed. Maybe you need to create a unique id combining host and dir. I would recommend that you ask a new question saying what you want to do, what you did and what is not working, righ now is confusing to understand as everything is split in multiple places. – leandrojmp Jun 11 '21 at 15:01
  • Will do as you advised on making a unique Id with host and dir and see the outcome. And sure, I will ask about question explaining the issue. But this time, without being the curious cat! – Number13 Jun 11 '21 at 15:38
0
input { 
    file    {
        path => "/home/dev*/status.log*"
        exclude => "status.log.10"
        start_position => "beginning"
        sincedb_path => "/dev/null"
#       sincedb_path => "/home/dev/db/devdb"
        file_sort_by => "path"
        file_sort_direction => "desc"
        }
}



output
{
stdout { codec => rubydebug }
}

filter {


if [path] =~ "dev1" 
{
mutate
{
replace => { "host" => "dev1" }
}
}
else if [path] =~ "dev2" 
{
mutate
{
replace => { "host" => "dev2" }
}
}
else if [path] =~ "dev3" 
{
mutate
{
replace => { "host" => "dev3" }
}
}
else if [path] =~ "dev4" 
{
mutate
{
replace => { "host" => "dev4" }
}
}


if [message] =~ "devManager"
{
grok
{
match => { "message" => "(?<logtime>%{DAY} %{MONTH} %{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}).*= %{BASE10NUM:status}" }
                    
}
date
{
match =>  [ "logtime", "EEE MMM dd HH:mm:ss.SSS" ] 
}

    if [status] == "0" {
    mutate
    {
      update => { "status" => "down" }
    }
    }
    else if [status] == "1" {
    mutate
    {
      update => { "status" => "up" }
    }
    }

mutate
{
add_tag => [ "%{status}" ]
}

elapsed
{
start_tag => "up"
end_tag => "down"
unique_id_field => "host"
timeout => 86400
}

elapsed
{
start_tag => "down"
end_tag => "up"
unique_id_field => "host"
timeout => 86400
}

if "up" in [tags] and [host]
{
mutate
{
add_field => { "host_down" => "%{elapsed_time}" }
}
mutate
{
convert =>
{
"host_down" => "float"
}
}
}



else if "down" in [tags] and [host]
{
mutate
{
add_field => { "host_up" => "%{elapsed_time}" }
}
mutate
{
convert =>
{
"host_up" => "float"
}
}
}


mutate
{
rename => { 
"status" => "%{host}_status" 
"host_up" => "%{host}_up"
"host_down" => "%{host}_down"
}
remove_field => [ "info" , "@version"    ]
}
}        
     
else { drop { } }

Here is the conf file which i am using with one worker. the path - "dev*" has dev1 to dev12 folders, which are to be read from.

log sample is as below;

/dev/status.log
Wed Jun 09 22:26:37.296  devManager: status = 1
Wed Jun 09 23:09:40.191  devManager: status = 0
Wed Jun 09 23:10:17.064  devManager: status = 0
Wed Jun 09 23:11:14.692  devManager: status = 1

@leandrojmp

Number13
  • 3
  • 4
0

I am thinking of another way. May be a little extra step to bring all my devs data to be ordered by one pipeline. Here's the process;

  1. Make pipeline to capture all the status = 0/1 from all devices, lets name it grabber.conf
  2. grabber.conf will do grok, and output to a status.log on disk. This status.log will have all the data on the status from all devs.
  3. grabber.conf will have input { exec { command => "sort...." } } . This sort will be directed to the status.log to sort all the dates in order and save itself. ( the interval for exec is trivial ).
  4. Make another pipeline to just execute elapsed filter on all the status from devs by adding tags to them as start and end. Lets name it durations.conf
  5. So, no matter what order of dates the logs are gathered and saved on status.log, the sort command will arrange them in order and make it ready for durations.conf to do its elapsed calculations.

Another risk of duplication of data can be avoided by adding fingerprint to the durations.conf.

Let me know if this could be an alternative solution to my query.

Number13
  • 3
  • 4