ConnectionError ECONNREFUSED connecting to OpenSearch on localhost from asp.net core app

Question

I have an OpenSearch domain on Amazon OpenSearch Service. I use it for my asp.net core app. I now want to start using localhost by running the default docker-compose file from the quickstart docs, instead of using the Amazon OpenSearch Service (while in development). I run the docker-compose file and it is all good. But when I run my asp.net app I start to see this come through in the docker terminal:

"error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 172.18.0.4:9200"

EDIT: I also see this a lot:

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record

I'm pretty sure the only thing I should be changing in my asp.net app is these connection strings in appsettings.json:

  "elasticEndpoint": "https://search-xxxxxxxxxxxxxxxxx.ap-southeast-2.es.amazonaws.com",
  "elasticIndexName": "vepo",
  "elasticUsername": "xxx",
  "elasticPassword": "xxx"

So I changed it to:

  "elasticEndpoint": "http://localhost:9200",
  "elasticIndexName": "vepo",
  "elasticUsername": "admin",
  "elasticPassword": "admin"

As far as I know that should be all that is needed, anyone know if I'm missing something or did it wrong?

I will now dump a massive amount of info below that you probably don't need but I wrote it before I found the ECONNREFUSED error and it may be useful, who knows:

Every time I start the asp.net app it deletes the default index and recreates it, then recreates every other index and populates it from my Postgres database.

This is where i create the OpenSearch client:

public class SearchService : ConfigurableSearchService, ISearchService
{
    private readonly string _defaultIndex;
    private readonly bool _refreshOnUpdate;
    private OpenSearchClient _client;

    public static readonly Dictionary<Type, string> SearchIndexMappings = new Dictionary<Type, string>
    {
        {typeof(VgnItmSearchDto), "vgnitmsearchdto"},
        {typeof(VgnItmEstSearchDto), "vgnitmestsearchdto"},
    };

    public string DefaultIndex { get => _defaultIndex; }

    public SearchService(
        string uri,
        string username,
        string password,
        string defaultIndex,
        bool forceInMemory = false,
        bool refreshOnUpdate = false)
       : base(SearchService.SearchIndexMappings)
    {
        ConnectionSettings connectionSettings;
        _defaultIndex = defaultIndex;
        _refreshOnUpdate = refreshOnUpdate;

        var node = new Uri(uri);
        IConnectionPool pool = new SingleNodeConnectionPool(node);

        connectionSettings = forceInMemory
                           ? new ConnectionSettings(pool, new InMemoryConnection())
                           : new ConnectionSettings(pool).BasicAuthentication(username, password);

        connectionSettings = connectionSettings
            .DefaultIndex(defaultIndex);

        IConnectionSettingsValues settings = connectionSettings;

        foreach (var mapping in SearchService.SearchIndexMappings)
        {
            settings.DefaultIndices.Add(mapping.Key, mapping.Value);
        }

        #if (DEBUG)
        connectionSettings.DisableDirectStreaming();
        #endif
        _client = new OpenSearchClient(connectionSettings);
    }

And that is called in Startup.cs, ConfigureServices():

services.AddScoped<ISearchService, SearchService>(serviceProvider =>
{
    return new SearchService(
            Configuration["elasticEndpoint"],
            Configuration["elasticUsername"],
            Configuration["elasticPassword"],
            Configuration["elasticIndexName"]);
});

The Configuration is using this from appsettings.json, this was the AWS one:

  "elasticEndpoint": "https://search-xxxxxxxxxxxxxxxxx.ap-southeast-2.es.amazonaws.com",
  "elasticIndexName": "xxx",
  "elasticUsername": "xxx",
  "elasticPassword": "xxx"

So I changed it to:

  "elasticEndpoint": "http://localhost:9200",
  "elasticIndexName": "vepo",
  "elasticUsername": "admin",
  "elasticPassword": "admin"

At the bottom of Configure() I tell the searchService to reindex all:

searchIndexService.ReIndex(context, SearchIndexes.All);

To run OpenSearch locally, I have the default docker-compose file from the quick start docs:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

I thought that if I run OpenSearch locally with docker-compose up, then I run my asp.net app, that it would then create my indexes in my local version of OpenSearch viewable at http://localhost:5601/app/home#/. But it doesn't. The Create index code all runs without any exceptions.

But my indexes do not get added there. And when I perform a search, in my asp.net app I get:

System.IO.IOException: The response ended prematurely. at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

And the OpenSearch docker terminal shows:

io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record

What am I missing that seems to be not connecting my asp.net app to my localhost version of OpenSearch?

Answer 1

first: If you are running your app from a docker container not use localhost, use the local IP of the host machine.

---

I have read about the issue and I found that opensearch uses tls certificates for all connections: node-node, and client-node.

https://opensearch.org/docs/latest/security/configuration/tls/

So you could create and install the certificates by yourself to allow your app (client) to connect to the local cluster (node). But if you think is not necessary to install these certificates in the local deployment, I suggest you avoid the SSL verification.

---

before all, if you have curl (bash terminal) execute this command:

   curl https://admin:admin@localhost:9200 -k

the -k asks curl not make ssl validation and admin:admin is username:password. And if the command works this solution will work for you.

Disabling the Security plugin in Opensearch

in docker-composer.yaml add "DISABLE_SECURITY_PLUGIN=true" to the environments:

    ...
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligible to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - "DISABLE_SECURITY_PLUGIN=true"
    ...

and

    ...
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_SECURITY_PLUGIN=true"
    ...

now you can use http instead https:

curl http://admin:admin@localhost:9200

Disabling SSL on your app

I'm not good at asp net core but I made a test disabling SSL validation and it worked(using HttpClient and your composer.yaml file).

    var handler = new HttpClientHandler();
    handler.ClientCertificateOptions = ClientCertificateOption.Manual;
    handler.ServerCertificateCustomValidationCallback = 
        (httpRequestMessage, cert, cetChain, policyErrors) =>
    {
        return true;
    };
    string url = "https://localhost:9200";
    HttpClient client = new HttpClient(handler);
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", "YWRtaW46YWRtaW4=");
    using HttpResponseMessage response = await client.GetAsync(url);
    var jsonResponse = await response.Content.ReadAsStringAsync();
    Console.WriteLine($"{jsonResponse}\n");

YWRtaW46YWRtaW4= is a base64 string encode from "user:password" ("admin:admin")

the output:

{
  "name" : "opensearch-node1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "mHWmnO2ORguqfe4F-0nJrQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.8.0",
    "build_type" : "tar",
    "build_hash" : "db90a415ff2fd428b4f7b3f800a51dc229287cb4",
    "build_date" : "2023-06-03T06:24:25.112415503Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

As you can see it works, so I think you should look into OpenSearch.Client package to know how to disable SSL and if it is possible, you have a solution.

Nginx solution

Also, you can see the issue from outside of your application: your app needs an endpoint and doesn't need to concern about certificates. So we can provide an endpoint ready to consume without SSL certificates.

How ? Using an Nginx container as a reverse proxy for "https://opensearch-node1:9200". Nginx proxy by default does not validate SSL certificates, then you could make a request to http://nginx-service:80 and it will work.

steps:

1. create a nginx.conf file in the same folder of docker-compose.yaml and paste it into it.

server {
    listen 80;
    listen [::]:80;

        
    location / {
        proxy_pass https://opensearch-node1:9200;
    }
}

2. add this service to your docker-composer.yaml

  nginx-proxy:
    image: nginx:latest
    container_name: nginx-proxy
    ports:
      - 80:80
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    networks:
      - opensearch-net

3. start your containers:

docker-compose up

4. test if the nginx works with curl (of another tool like Postman):

curl http://admin:admin@localhost

5. change the endpoint in your asp net core app, for example:

    public class Student
    {
        public int Id { get; init; }
        public string FirstName { get; init; }
        public string LastName { get; init; }
        public int GradYear { get; init; }
        public double Gpa { get; init; }
    }

    string url = "http://localhost";
    Console.WriteLine(url);
    var node = new Uri(url);
    var config = new ConnectionSettings(node).BasicAuthentication("admin", "admin").DefaultIndex("students");
    var client = new OpenSearchClient(config);
    Console.WriteLine($"{config}\n");
    var student = new Student { Id = 100, FirstName = "Paulo", LastName = "Santos", Gpa = 3.93, GradYear = 2021 };
    var response = client.Index(student, i => i.Index("students")); 
    Console.WriteLine($"{response}\n");

Valid OpenSearch.Client response built from a successful (201) low level call on PUT: /students/_doc/100

6. read indexes:

curl http://admin:admin@localhost/students/_doc/100

{"_index":"students","_id":"100","_version":2,"_seq_no":1,"_primary_term":2,"found":true,"_source":{"id":100,"firstName":"Paulo","lastName":"Santos","gradYear":2021,"gpa":3.93}}

I hope it helps you.