0

I've been trying to run a very simple SQL query from logstash.yml file and making a query to AWS opensearch.

logstash.yml file:

input {
    jdbc {
            jdbc_connection_string => "jdbc:postgresql://abcxyz.ap-south-1.rds.amazonaws.com:5432/k***"
            jdbc_user => "k***"
            jdbc_password => "p**********s"
            jdbc_driver_library => "/path/to/postgresql-42.3.0.jar"
            jdbc_driver_class => "org.postgresql.Driver"
            jdbc_paging_enabled => true
            tracking_column => "unix_ts_in_secs"
            use_column_value => true
            tracking_column_type => "numeric"
            schedule => "*/5 * * * * *"

            statement => 'SELECT name FROM "User_profile"'
            # statement => 'SELECT * FROM "Video_Videos"'
    }
}
filter {
    mutate {
        copy => { "id" => "[@metadata][_id]"}
        remove_field => ["id", "@version", "unix_ts_in_secs"]
    }
}
output {
    opensearch {
            hosts => "https://xyz.ap-south-1.es.amazonaws.com:443"
            user => "k**h"
            password => "E****"
            index => "rdbms_sync_idx"
            document_id => "%{[@metadata][_id]}"
            ssl => true
            ssl_certificate_verification => false
    }
}

but whenever I try to run: ./logstash -f /etc/logstash/conf.d/cp_logstash.yml from my command terminal,

I get the following error trace[updated]:

Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-12-24 04:19:02.723 [main] runner - Starting Logstash {"logstash.version"=>"7.16.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[WARN ] 2021-12-24 04:19:03.460 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2021-12-24 04:19:03.484 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[FATAL] 2021-12-24 04:19:03.487 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
    at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Not sure how to fix these errors. Any suggestions ?

thanks,

Saravanan G
  • 581
  • 1
  • 9
  • 26
Naveen
  • 1
  • 1
  • If you are using the Elastic version of logstash then the opensearch output is not bundled with it. Are you using Elastic logstash or OSS logstash? – Badger Dec 23 '21 at 17:39
  • @Badger how can i check which one am i using ? from logstash yml file, i can see opensearch keyword in the 'output' part. – Naveen Dec 24 '21 at 02:07
  • updated error logs – Naveen Dec 24 '21 at 04:42

1 Answers1

0

The problem is mentioned on this line:

Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

This means you're running another instance of Logstash on the same host that shares the same data directory and that's not allowed.

Simply make sure that each instance has a different path.data setting in their respective configurations.

Val
  • 207,596
  • 13
  • 358
  • 360
  • sudo systemctl status logstash sudo systemctl stop logstash ./logstash -f /etc/logstash/conf.d/logstash.yml getting same error after these 3 commands as well – Naveen Dec 24 '21 at 07:07
  • how can i change this ```path.data``` setting ? – Naveen Dec 24 '21 at 07:09
  • Here is a good way to achieve this: https://discuss.elastic.co/t/solved-multiple-instances-for-logstash/65539/2?u=theuntergeek – Val Dec 24 '21 at 07:17