0

I have been able to create a python application that connects to Amazon OpenSearch using the python OpenSearch library and the necessary credentials to sign in to the OpenSearch API.

Python Code to connect to OpenSearch Client:

import streamlit as st
import pandas as pd
import json
from opensearchpy import OpenSearch

host = 'localhost'
port = 9200
auth = ('admin', 'admin') # For testing only. Don't store credentials in code.

# Create the client with SSL/TLS enabled, but hostname verification disabled.
client = OpenSearch(
    hosts = [{'host': host, 'port': port}],
    http_compress = True, # enables gzip compression for request bodies
    http_auth = auth,
    verify_certs = False,

)

Here is my code to create indices and load data to my indices:

index_name = 'news_articles'
# Delete the index.
response = client.indices.delete(
    index = index_name
)

print('\nDeleting index:')
print(response)

# Create an index with non-default settings.

index_body = {
  'settings': {
    'index': {
      'number_of_shards': 5
    }
  }
}

response = client.indices.create(index_name, body=index_body)
print('\nCreating index:')
print(response)


# Add a document with news articles content to the index.
news_articles = pd.read_csv(r'News.csv')

news_articles = news_articles.fillna('')

for i in range(0,len(news_articles['Article Title'])):
    doc_1 = {'title':news_articles['Article Title'][i], 'text':news_articles['Article Text'][i], 'publishing_date':news_articles['Article Date'][i], 'author':news_articles['Article Author'][i]}
    response = client.index(
    index = index_name,
    body = doc_1,
    id = i,
    refresh = True)
    print('\nAdding document:')
    print(response)

This works perfect when I ran it on my local computer. The issues rise when I convert the application to a docker image and try to connect the streamlit application to Amazon Opensearch.

Here is my requirements.txt file for the streamlit application:

opensearch_py==1.0.0
pandas==1.2.4
streamlit==1.0.0

Here is my dockerfile to build the Streamlit python app docker image:

FROM python:3.8.8

WORKDIR /streamlitApps

COPY ./requirements.txt /requirements.txt
COPY News.csv ./News.csv

RUN apt update && \
    apt install --no-install-recommends -y build-essential gcc && \
    apt clean && rm -rf /var/lib/apt/lists/* && \
    pip3 install --no-cache-dir --upgrade pip setuptools && \
    pip install -r /requirements.txt

EXPOSE 8501

COPY . /streamlitApps

ENTRYPOINT ["streamlit", "run"]

CMD ["sentiment_analysis2.py"]

After building the image and I try to run the image on docker using the command in Terminal on Visual Studio Code:

docker run -p 8501:8501 sentiment_analysis2:latest

Full error log from Docker:

2021-12-29 17:21:09.312 DELETE http://localhost:9200/news_articles [status:N/A request:0.005s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/connection/http_urllib3.py", line 256, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f85f42ab160>: Failed to establish a new connection: 
[Errno 111] Connection refused
2021-12-29 17:21:09.327 DELETE http://localhost:9200/news_articles [status:N/A request:0.006s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/connection/http_urllib3.py", line 256, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f85f4208df0>: Failed to establish a new connection: 
[Errno 111] Connection refused
2021-12-29 17:21:09.330 DELETE http://localhost:9200/news_articles [status:N/A request:0.002s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/connection/http_urllib3.py", line 256, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f85e5b87c70>: Failed to establish a new connection: 
[Errno 111] Connection refused
2021-12-29 17:21:09.345 DELETE http://localhost:9200/news_articles [status:N/A request:0.013s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/connection/http_urllib3.py", line 256, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f85e5b94910>: Failed to establish a new connection: 
[Errno 111] Connection refused
2021-12-29 17:21:09.351 Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/connection/http_urllib3.py", line 256, in perform_request
    response = self.pool.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f85e5b94910>: Failed to establish a new connection: 
[Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/streamlit/script_runner.py", line 354, in _run_script
    exec(code, module.__dict__)
  File "/streamlitApps/sentiment_analysis2.py", line 20, in <module>
    response = client.indices.delete(
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/client/utils.py", line 177, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/client/indices.py", line 296, in delete
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/transport.py", line 410, in perform_request
    raise e
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/transport.py", line 373, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/site-packages/opensearchpy/connection/http_urllib3.py", line 271, in perform_request
    raise ConnectionError("N/A", str(e), e)
opensearchpy.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f85e5b94910>: Failed to establish 
a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f85e5b94910>: Failed to establish a new connection: [Errno 111] Connection refused)

2021-12-29 17:21:09.930 Trying to detect encoding from a tiny portion of (14) byte(s).
2021-12-29 17:21:09.931 ascii passed initial chaos probing. Mean measured chaos is 0.000000 %
2021-12-29 17:21:09.932 ascii should target any language(s) of ['Latin Based']
2021-12-29 17:21:09.933 ascii is most likely the one. Stopping the process.

  You can now view your Streamlit app in your browser.

  Network URL: http://172.17.0.2:8501
  External URL: http://199.166.186.1:8501
David Maze
  • 130,717
  • 29
  • 175
  • 215
Samuel Oliver Bedu-Annan
  • 1
  • 3
  • The Elasticsearch location will not be `localhost` in either Docker (`localhost` means "this container") or AWS (Elasticsearch will have its own AWS-provided hostname). Do you have some way to configure its location at deploy time? – David Maze Dec 29 '21 at 18:01
  • @DavidMaze Docker has Elasticsearch images that you can pull and connect with your python application. I am having a hard time doing that. The name of the elasticsearch docker container is `docker.elastic.co/elasticsearch/elasticsearch` and the corresponding OpenSearch docker container is `opensearchproject/opensearch`. Do I change the `localhost` to one of these names ? – Samuel Oliver Bedu-Annan Dec 29 '21 at 20:44
  • @DavidMaze I tried to change the `localhost` to `docker.elastic.co/elasticsearch/elasticsearch` or `opensearchproject/opensearch` but I get `opensearchpy.exceptions.ConnectionError: ConnectionError(: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(: Failed to establish a new connection: [Errno -2] Name or service not known)` . Please let me know if i can try anything else. – Samuel Oliver Bedu-Annan Dec 29 '21 at 20:58
  • Those sound like Docker image names, not DNS names. In a Compose-based setup, you should read through background material like [Networking in Compose](https://docs.docker.com/compose/networking/) to understand what host names are available to you. In an AWS environment, the OpenSearch instance should have its own host name that's specific to your environment. – David Maze Dec 29 '21 at 22:28

0 Answers0