Avoiding ElasticSearch error 503 Server Unavailable: Use of WaitForStatus

Question

When I start my program, I run the ElasticSearch Service and check if an Index exists and if there is any documents, let's say I just run the ES service and I have these two functions:

public ElasticClient getElasticSearchClient()
{
    ConnectionSettings connectionSettings = new Nest.ConnectionSettings(new Uri("http://localhost:9200"))
                                                    .DefaultIndex("myindex")
                                                    .DisableDirectStreaming();
    ElasticClient client = new ElasticClient(connectionSettings);
    //var health = client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)).Timeout(50));
    return client;
}

public void checkElasticsearchIndex()
{
    var client = getElasticSearchClient();

    var health = this.client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)));

    CountResponse count = client.Count<myobject>();

    if (!client.Indices.Exists("myindex").IsValid || count.Count == 0)
    {
        BulkWriteAllToIndexES(client);
    }
}

Inside the checkElasticsearchIndex function,

1. The count operation fails with the following error message:

   OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (503) Server Unavailable.. Call: Status code 503 from: GET /myindex/_count. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

2. The Health fails as well:

   OriginalException: Elasticsearch.Net.ElasticsearchClientException: Unable to connect to the remote server. Call: Status code unknown from: GET /_cluster/health/myindex?wait_for_status=yellow ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:9200

As you can see, I have tried the Cluster WaitForStatus, but it didn't work.

My question: is there any way to wait until client/cluster/nodes are ready and not get any exception?

Answer 1

It sounds like you're starting the Elasticsearch process at the same time as starting your program, but Elasticsearch takes longer than your program to be ready.

If that's the case, you may be interested in using the same abstractions that the .NET client uses for integration tests against Elasticsearch. The abstractions read output from the Elasticsearch process to know when it is ready, and block until this happens. They're available on an AppVeyor CI package feed (with plans to release them to Nuget in the future).

There are some examples of how to spin up a cluster with the abstractions. For single node, it would be something like

using System;
using Elastic.Managed.Configuration;
using Elastic.Managed.ConsoleWriters;
using Elastic.Managed.FileSystem;

namespace Elastic.Managed.Example
{
    class Program
    {
        static void Main(string[] args)
        {
            var version = "7.5.1";
            var esHome = Environment.ExpandEnvironmentVariables($@"%LOCALAPPDATA%\ElasticManaged\{version}\elasticsearch-{version}");

            using (var node = new ElasticsearchNode(version, esHome))
            {
                node.SubscribeLines(new LineHighlightWriter());
                if (!node.WaitForStarted(TimeSpan.FromMinutes(2))) throw new Exception();

                // do your work here
            }
        }
    }
}

This assumes that Elasticsearch 7.5.1 zip has been downloaded already, and exists at %LOCALAPPDATA%\ElasticManaged\7.5.1\elasticsearch-7.5.1. There are more complex examples of how to integrate this into tests with xUnit.

You can use the EphemeralCluster components to download, configure and run Elasticsearch

var plugins = new ElasticsearchPlugins(ElasticsearchPlugin.RepositoryAzure, ElasticsearchPlugin.IngestAttachment);
var config = new EphemeralClusterConfiguration("7.5.1", ClusterFeatures.XPack, plugins, numberOfNodes: 1);
using (var cluster = new EphemeralCluster(config))
{
    cluster.Start();

    var nodes = cluster.NodesUris();
    var connectionPool = new StaticConnectionPool(nodes);
    var settings = new ConnectionSettings(connectionPool).EnableDebugMode();
    var client = new ElasticClient(settings);

    Console.Write(client.CatPlugins().DebugInformation);
}