2

Official logstash elastic cloud module

Official doc for starting with

My logstash.yml looks like:

  cloud.id: "Test:testkey"
  cloud.auth: "elastic:password"

With 2 spaces in front and no space at end, within ""

This is all I have in logstash.yml and nothing else,
And I am getting:

[2018-08-29T12:33:52,112][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://myserverurl:12345/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://myserverurl:12345/'"}

And the my_config_file_name.conf looks like:

input{jdbc{...jdbc here... This works, as I see data in windows console}}
output {
    stdout { codec => json_lines }
    elasticsearch {
    hosts => ["myserverurl:12345"]
    index => "my_index"
    # document_id => "%{brand}"
    }

What I am doing is hitting bin/logstash on windows cmd,
It loads data from database that I have configured in input of conf file and then shows me error, I want to index my data from MySQL to elasticsearch on Cloud, I took 14 days trial and created a test index, for learning purpose as I later have to deploy it. My Pipeline looks like:

  - pipeline.id: my_id
    path.config: "./config/conf_file_name.conf"
    pipeline.workers: 1

If logs won't include senistive data, I can also provide them.
Basically I wan't to sync (schedule check) my MYSQL data with ElasticSearch on cloud i.e. AWS

Aashish Gahlawat
  • 409
  • 1
  • 7
  • 25

3 Answers3

2

Also if you created your API key in the web UI you will not be able to get the values needed to configure Logstash. You must to use the devtool console found at /app/dev_tools#/console with something like this:

POST /_security/api_key
{
  "name": "logstash"
}

of which the output is something like:


{
  "id": "<id value>",
  "name": "logstash",
  "api_key": "<api key>",
  "encoded": "<encoded api key>"
}

And in your logstash pipeline config you use the values like this:

output {
  elasticsearch {
    cloud_id => "<cloud id>"
    api_key => "<id value>:<api key>"
    data_stream => true
    ssl => true
  }
  stdout { codec => rubydebug }
}

Note the combined "api_key" value separated by ":". Also, you can find the "cloud id" under your "Deployments" menu option.

PålOliver
  • 2,502
  • 1
  • 23
  • 27
1

The output shall be:

elasticsearch {
    hosts => ["https://yourhost:yourport/"]
    user => "elastic"
    password => "password"
    # protocol => https
    # port => "yourport"
    index => "test_index"
    # document_id => "%{table_id}"

# - represent comments

as stated at: Configuring logstash with elastic cloud docs

The document provided while deploying app does not provide config for jdbc, jdbc as well need user and password even if defined in settings file i.e. logstash.yml

Aashish Gahlawat
  • 409
  • 1
  • 7
  • 25
0

I add the same issue in my dev environment. After scour hours on google, I understood by default, when you install Logstash, X-Pack is installed. In the doc https://www.elastic.co/guide/en/logstash/current/setup-xpack.html it is stated that

Blockquote

X-Pack is an Elastic Stack extension that provides security, alerting, monitoring, machine learning, pipeline management, and many other capabilities

Blockquote

As I don't need x-pack to run in my dev while I am streaming Elasticsearch, I had to disable it by setting ilm_enabled to false in the output of my indexation file configuration.

    output {
     elasticsearch { 
        hosts =>  [.. ]
        ilm_enabled => false
      }
    }

The link bellow may help https://discuss.opendistrocommunity.dev/t/logstash-oss-with-non-removable-x-pack/655/3

onlyme
  • 3,776
  • 2
  • 23
  • 17