1

Below is the script that I am using to access index data from elasticsearch in spark scala code. It is working perfectly. 

import com.sksamuel.elastic4s.http.{ElasticClient, ElasticProperties}
import com.sksamuel.elastic4s.http.ElasticDsl._
val esUrl = "http://" + <ip> + ":" + <port>
val client = ElasticClient(ElasticProperties(esUrl))
val response = client.execute(
        deleteIn(Index/Type).by(matchQuery(indexField, values))
      )

Now I have configured ES as SSL and trying to access same data but it is not working since I need to pass SSL details here. So I have passed below url but it is giving invalid url error.

https://user:pass@<ip>:<port>?ssl=true"

or

https://user:pass@<ip>:<port>"

Could you please help me to change this script so that it can also work with SSL configured ES?

I have also tried to use HttpClient instead of ElasticClient but it is not working and giving weird error like below:

scala> import com.sksamuel.elastic4s.http.HttpClient
import com.sksamuel.elastic4s.http.HttpClient

scala> val esUrl = "elasticsearch://user:pass@<ip>:<port>?ssl=true"
esUrl: String = elasticsearch://user:pass@<ip>:<port>?ssl=true

scala> val client = HttpClient(ElasticsearchClientUri(esUrl))
    <console>:43: error: not found: value HttpClient

I want to use same ElasticClient for it. Please let me know if I have missed anything.

Please note: I am using below version:

Spark - 2.4.3

ElasticSearch - 6.2.4

elastic4s - 6.3.3

scala - 2.11

Nikhil Suthar
  • 2,289
  • 1
  • 6
  • 24

1 Answers1

1

I have done few research for elastic4s and conclude that it support SSL but does not support Authentication (using Search Guard). So I have written my own Scala Method to perform all required Operation over Elastic Search. It is working perfectly.

My GitHub link is below:

https://medium.com/@n.suthar.online/scalaes-apache-spark-and-elasticsearch-connector-d6bc927f8c51

Nikhil Suthar
  • 2,289
  • 1
  • 6
  • 24
  • Can you share your code for authentication. and How to create template using json file haveing (mappings, setting). – user1668782 Aug 31 '19 at 13:08
  • Sorry for delay , Please check link: https://github.com/NikhilSuthar/ScalaES – Nikhil Suthar Oct 10 '19 at 11:01
  • @Nikk Your link is not working. Can you. please share your code. – Basant Jain May 21 '20 at 07:52
  • @Nikk I want to pass ssl truststore certificate path and ssl truststore password along with authentication. Can you please help me with this. – Basant Jain May 21 '20 at 07:56
  • @BasantJain Sorry I will not be able to provide you code. But I have published one blog on medium that can help you. I have updated above link please check it. You can pass user name and password directly and keep SSL flag true. For certificate, you need to install it in your Java Keytool. Go through blog, that will help you – Nikhil Suthar May 21 '20 at 08:03