-1

I have 16 conf files and all of them scheduled to run every day at 09:05 am. Today these files could not run at intended time. After i fix the problem tried to restart logstash but conf files are not able to generate indices.

Example dash_KPI_1.conf file:

input {
 jdbc {
    jdbc_driver_library => "/var/OJDBC-Full/ojdbc6.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "jdbc:oracle:thin:@a/b"
    jdbc_user => "KIBANA"
    jdbc_password => "pass"
    statement => "
                SELECT  /*+ PARALLEL(16) */
                * from
           dual"
  # jdbc_paging_enabled => "true"
  # jdbc_page_size => "50000"
    type => "dash_kpi_1"
    schedule => "05 09 * * *"
  }
}
output {  if [type]=="dash_kpi_1"{
#  stdout { codec => rubydebug }
   elasticsearch {
             hosts => ["http://XX.XX.XX.XXX:9200","http://XX.XX.XX.XXX:9200","http://XX.XX.XX.XXX:9200"]
             index => "dash_kpi_1-%{+YYYY.ww}"
             user => "elastic"
             password => "pass2"
                      }
 }
}

How i start and stop logstash:

systemctl stop logstash.service
systemctl start logstash.service -r

What i have tried:

/usr/share/logstash/bin/logstash -f dash_KPI_1.conf

How can i restart these 16 conf files and make them generate indices as intended in the first place ?

Santosh Bhosle
  • 123
  • 3
  • 15
ismetguzelgun
  • 1,090
  • 8
  • 16
  • Have you tried refreshing index list in Kibana ? – Santosh Bhosle Aug 17 '20 at 11:58
  • No i did not refresh them because there are already generated indices on kibana. As you can see from my conf file every day this conf file produces an index as named index => "dash_kpi_1-%{+YYYY.ww}". So there is nothing to restart from kibana last index is yesterday's index. – ismetguzelgun Aug 17 '20 at 12:07
  • What do you have in your logstash and elasticsearch logs? If logstash couldn't create your index you probably have something in your logs. – leandrojmp Aug 17 '20 at 13:29
  • Yes there was a problem and i fixed it. But time passed and i need to restart my daily schedule again. @leandrojmp – ismetguzelgun Aug 17 '20 at 13:36
  • Did you change your pipeline when you restarted? You have a `schedule` line in your input plugin set to `09:05`, you need to change the time before restarting. – leandrojmp Aug 17 '20 at 13:53
  • I know i need to do this. But doing this one by one 16 conf files(or even if i write a .sh for the sole purpose of automating this process it would still be a waste of time) and after restart resetting them all together makes no sense. This is why i am asking for your advices. Maybe there is a way which accomplishes this whole job much faster that i was not aware of ? @leandrojmp – ismetguzelgun Aug 17 '20 at 14:09
  • You can change your config to use an [environment variable](https://www.elastic.co/guide/en/logstash/current/environment-variables.html) as your scheduler, something like `schedule => ${SCHEDULE_TIME}` and set the environment variable with the schedule pattern that you want, this way you would need to change only the variable. But you will still need to edit all the files, at least one more time. – leandrojmp Aug 17 '20 at 14:26
  • I know i can do this but i would like to avoid all those steps. Thank you for your time and help. @leandrojmp – ismetguzelgun Aug 17 '20 at 14:32

1 Answers1

0

I see you are creating index weekly. If you want to create it daily, you need to change the index pattern to "dash_kpi_1-%{+YYYY.MM.dd}".

Santosh Bhosle
  • 123
  • 3
  • 15
  • Nope this is not i want. I want to know how to restart a conf file in logstash which normally runs once a day. – ismetguzelgun Aug 17 '20 at 12:26
  • Below link talk about reloading the config files. May be you can take a look at it once.
    [Reloading-config](https://www.elastic.co/guide/en/logstash/current/reloading-config.html)
    [logstash setting file](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html)     – Santosh Bhosle Aug 17 '20 at 13:03
  • Unfortunately that does not helpful. – ismetguzelgun Aug 17 '20 at 13:27