1

I have an AWS ElasticSearch domain configured. I have installed LogStash in an EC2 and it used to push data into the ES domain and I could view them in the Kibana Discover. Since yesterday, Discover does not load properly. When I click the Discover in Home screen, it loads for a while and then stops but the screen is all blank.

Inside the EC2 server, when I check LogStash status (service logstash status), I see multiple errors and warnings.

logstash[8997]: [INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash[8997]: [INFO ][org.logstash.beats.Server][main][e4620d298b3dbd13b2a68a1f1fcd717a9e12c8414d40373b93cefd5adeacb1c9] Starting server on port: 5044
logstash[8997]: [WARN ][logstash.outputs.amazonelasticsearch][main][43b973399a83b46ed67e936feeb816720e9ea3d1d511c25b9079ddc2091ccef7] Marking url as dead. Last error: [LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://_esdomain_.es.amazonaws.com:443/][Manticore::ClientProtocolException] _esdomain_.es.amazonaws.com:443 failed to respond {:url=>https://_esdomain_.es.amazonaws.com:443/, :error_message=>"Elasticsearch Unreachable: [https://_esdomain_.es.amazonaws.com:443/][Manticore::ClientProtocolException] _esdomain_.es.amazonaws.com:443 failed to respond", :error_class=>"LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash[8997]: [ERROR][logstash.outputs.amazonelasticsearch][main][43b973399a83b46ed67e936feeb816720e9ea3d1d511c25b9079ddc2091ccef7] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://_esdomain_.es.amazonaws.com:443/][Manticore::ClientProtocolException] _esdomain_.es.amazonaws.com:443 failed to respond", :class=>"LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>4}
logstash[8997]: [INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://_esdomain_.es.amazonaws.com:443/, :path=>"/"}
logstash[8997]: [WARN ][logstash.outputs.amazonelasticsearch][main] Restored connection to ES instance {:url=>"https://_esdomain_.es.amazonaws.com:443/"}
logstash[8997]: [WARN ][logstash.outputs.amazonelasticsearch][main][43b973399a83b46ed67e936feeb816720e9ea3d1d511c25b9079ddc2091ccef7] Marking url as dead. Last error: [LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://_esdomain_.es.amazonaws.com:443/][Manticore::SocketException] Broken pipe (Write failed) {:url=>https://_esdomain_.es.amazonaws.com:443/, :error_message=>"Elasticsearch Unreachable: [https://_esdomain_.es.amazonaws.com:443/][Manticore::SocketException] Broken pipe (Write failed)", :error_class=>"LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash[8997]: [ERROR][logstash.outputs.amazonelasticsearch][main][43b973399a83b46ed67e936feeb816720e9ea3d1d511c25b9079ddc2091ccef7] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://_esdomain_.es.amazonaws.com:443/][Manticore::SocketException] Broken pipe (Write failed)", :class=>"LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>16}
logstash[8997]: [INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://_esdomain_.es.amazonaws.com:443/, :path=>"/"}
logstash[8997]: [WARN ][logstash.outputs.amazonelasticsearch][main] Restored connection to ES instance {:url=>"https://_esdomain_.es.amazonaws.com:443/"}

I cannot figure out whether they're related to the Kibana Discover's unresponsiveness or not.

Sandun
  • 395
  • 2
  • 10
  • 25
  • Did you check elasticsearch logs? It seems that you could have some problems with elasticsearch as both Kibana and Logstash are not working. Check your elasticsearch logs. – leandrojmp Jun 11 '21 at 04:25
  • @leandrojmp I am using the AWS ElasticSearch service. So I cannot connect to any ElasticSearch server and check its log. There's a tab called 'Logs' on AWS ES dashboard, although I have to manually activate them. – Sandun Jun 11 '21 at 06:17
  • Well, you seem to be having problems with elasticsearch as per your logstash logs and your unresponsive Kibana, so you will need a way to see what is happening with your elasticsearch services. I don't know if this Logs tab will help, never used the AWS Elasticsearch, but according to the [documentation](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createdomain-configure-slow-logs.html) you will need to enable it. – leandrojmp Jun 11 '21 at 13:09

1 Answers1

0

What worked for me was instaling and starting Kibana locally on EC2.

Todor Sidorov
  • 1
  • 1