I try to configure the Logstash Output OpenSearch plugin to index data in OpenSearch 2.3.0 provisioned by AWS. The access to the AWS OpenService is controlled by IAM. I followed this configuration sample and built the following configuration:
output {
opensearch {
hosts => ["vpc-my-opensearch-domain-XXXXXXXX.us-east-1.es.amazonaws.com:443"]
auth_type => {
type => 'aws_iam'
aws_access_key_id => 'MY_ACCESS_KEY_ID'
aws_secret_access_key => 'MY_ACCESS_SECRET_KEY'
region => 'us-east-1'
}
index => "program_search_01"
document_id => "%{ruleProgramId}-%{ruleId}"
routing => "%{[programRulesRelationship][parent]}"
template_name => "my-template"
template => "/usr/share/logstash/pipeline/template/my-template.json"
manage_template => true
template_overwrite => true
}
}
After starting Logstash I am getting the following logs:
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.wire ][programAvailability] http-outgoing-25053 << "HTTP/1.1 400 Bad Request[\r][\n]"
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.wire ][programAvailability] http-outgoing-25053 << "Server: awselb/2.0[\r][\n]"
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.wire ][programAvailability] http-outgoing-25053 << "Date: Tue, 07 Feb 2023 05:00:05 GMT[\r][\n]"
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.wire ][programAvailability] http-outgoing-25053 << "Content-Type: text/html[\r][\n]"
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.wire ][programAvailability] http-outgoing-25053 << "Content-Length: 220[\r][\n]"
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.wire ][programAvailability] http-outgoing-25053 << "Connection: close[\r][\n]"
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.wire ][programAvailability] http-outgoing-25053 << "[\r][\n]"
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.headers ][programAvailability] http-outgoing-25053 << HTTP/1.1 400 Bad Request
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.headers ][programAvailability] http-outgoing-25053 << Server: awselb/2.0
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.headers ][programAvailability] http-outgoing-25053 << Date: Tue, 07 Feb 2023 05:00:05 GMT
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.headers ][programAvailability] http-outgoing-25053 << Content-Type: text/html
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.headers ][programAvailability] http-outgoing-25053 << Content-Length: 220
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.headers ][programAvailability] http-outgoing-25053 << Connection: close
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][programAvailability] http-outgoing-25053: Close connection
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.impl.execchain.MainClientExec][programAvailability] Connection discarded
[2023-02-07T05:00:05,306][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][programAvailability] Connection released: [id: 25053][route: {}->http://vpc-my-opensearch-domain-XXXXXXXXXX-east-1.es.amazonaws.com:443][total available: 0; route allocated: 0 of 100; total allocated: 0 of 1000]
According to the logs, the plugin use http instead of https. I suspect this to be a root cause. If yes, can anyone tell me what I am missing in my plugin configuration?
I am running logstash as a docker container, the existing docker file looks like this:
FROM opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.3
LABEL Name=cas-logstash
COPY mssql-jdbc-9.4.0.jre11.jar /usr/share/logstash/drivers/
USER root
RUN chown -R logstash:root /usr/share/logstash/drivers
RUN sed -i 's/path = "#{template_endpoint}\/#{name}"/path = "\/#{template_endpoint}\/#{name}"/' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client.rb
USER logstash
