I am trying to establish a connection between my Confluent Kafka cluster and Elasticsearch cluster using the ES service sink connector, both of which are managed services.
I have set up a PrivateLink between my ES cluster and AWS environment following Elastic's documentation, and I am able to get a response when using this URL: https://user:password@cluster-id.vpce.eu-west-1.aws.elastic-cloud.com
The problem arises when I try to create the sink connector. I have a peering connection between Kafka and AWS, and all the routes and security groups of the VPC endpoint are configured properly. However, it seems that the connector is unable to resolve the endpoint, and rightfully so because the Kafka's VPC not associated with the private hosted zone. Here is the error that I am receiving:
connection.url: Could not connect to Elasticsearch. Error message: java.util.concurrent.ExecutionException: java.net.UnknownHostException: cluster-id.vpce.eu-west-1.aws.elastic-cloud.com: Name or service not known connection.url: Failed to create client to verify connection. java.util.concurrent.ExecutionException: java.net.UnknownHostException: cluster-id.vpce.eu-west-1.aws.elastic-cloud.com connection.username: Could not authenticate the user. Check the 'connection.username' and 'connection.password'. Error message: java.util.concurrent.ExecutionException: java.net.UnknownHostException: cluster-id.vpce.eu-west-1.aws.elastic-cloud.com: Name or service not known connection.password: Could not authenticate the user. Check the 'connection.username' and 'connection.password'. Error message: java.util.concurrent.ExecutionException: java.net.UnknownHostException: cluster-id.vpce.eu-west-1.aws.elastic-cloud.com: Name or service not known
I would appreciate any opinions on how I can resolve this issue if someone has had a similar use case. Specifically, connecting Kafka using VPC peering connection to the Elasticsearch cluster with PrivateLink.
I tried to find out documentation regrading my use case and didn’t find any.
