Fields are empty when doing GET in elastic4s

Question

I'm trying to implement a service in my play2 app that uses elastic4s to get a document by Id.

My document in elasticsearch:

curl -XGET 'http://localhost:9200/test/venues/3659653'

{
    "_index": "test",
    "_type": "venues",
    "_id": "3659653",
    "_version": 1,
    "found": true,
    "_source": {
        "id": 3659653,
        "name": "Salong Anna och Jag",
        "description": "",
        "telephoneNumber": "0811111",
        "postalCode": "16440",
        "streetAddress": "Kistagången 12", 
        "city": "Kista",
        "lastReview": null,
        "location": {
            "lat": 59.4045675,
            "lon": 17.9502138
        },
        "pictures": [],
        "employees": [],
        "reviews": [],
        "strongTags": [
            "skönhet ",
            "skönhet ",
            "skönhetssalong"
        ],
        "weakTags": [
            "Frisörsalong",
            "Frisörer"
        ],
        "reviewCount": 0,
        "averageGrade": 0,
        "roundedGrade": 0,
        "recoScore": 0
    }
}

My Service:

@Singleton
class VenueSearchService extends ElasticSearchService[IndexableVenue] {

  /**
   * Elastic search conf
   */
  override def path = "test/venues"

  def getVenue(companyId: String) = {
    val resp = client.execute(
      get id companyId from path
    ).map { response =>
        // transform response to IndexableVenue
        response 
    }
    resp
  }

If I use getFields() on the response object I get an empty object. But if I call response.getSourceAsString I get the document as json:

{

    "id": 3659653,
    "name": "Salong Anna och Jag ",
    "description": "",
    "telephoneNumber": "0811111",
    "postalCode": "16440",
    "streetAddress": "Kistagången 12",
    "city": "Kista",
    "lastReview": null,
    "location": {
        "lat": 59.4045675,
        "lon": 17.9502138
    },
    "pictures": [],
    "employees": [],
    "reviews": [],
    "strongTags": [
        "skönhet ",
        "skönhet ",
        "skönhetssalong"
    ],
    "weakTags": [
        "Frisörsalong",
        "Frisörer"
    ],
    "reviewCount": 0,
    "averageGrade": 0,
    "roundedGrade": 0,
    "recoScore": 0
}

As you can se the get request omits info:

"_index": "test",
    "_type": "venues",
    "_id": "3659653",
    "_version": 1,
    "found": true,
    "_source": {}

If I try to do a regular search:

def getVenue(companyId: String) = {
    val resp = client.execute(
      search in "test"->"venues" query s"id:${companyId}"
      //get id companyId from path
    ).map { response =>
        Logger.info("response: "+response.toString)
    }
    resp
  }

I get:

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 1,
        "hits": [
            {
                "_index": "test",
                "_type": "venues",
                "_id": "3659653",
                "_score": 1,
                "_source": {
                    "id": 3659653,
                    "name": "Salong Anna och Jag ",
                    "description": "",
                    "telephoneNumber": "0811111",
                    "postalCode": "16440",
                    "streetAddress": "Kistagången 12",
                    "city": "Kista",
                    "lastReview": null,
                    "location": {
                        "lat": 59.4045675,
                        "lon": 17.9502138
                    },
                    "pictures": [],
                    "employees": [],
                    "reviews": [],
                    "strongTags": [
                        "skönhet ",
                        "skönhet ",
                        "skönhetssalong"
                    ],
                    "weakTags": [
                        "Frisörsalong",
                        "Frisörer"
                    ],
                    "reviewCount": 0,
                    "averageGrade": 0,
                    "roundedGrade": 0,
                    "recoScore": 0
                }
            }
        ]
    }
}

My Index Service:

trait ElasticIndexService [T <: ElasticDocument] {

  val clientProvider: ElasticClientProvider
  def path: String

  def indexInto[T](document: T, id: String)(implicit writes: Writes[T]) : Future[IndexResponse] = {
    Logger.debug(s"indexing into $path document: $document")
    clientProvider.getClient.execute {
      index into path doc JsonSource(document) id id
    }
  }


}

case class JsonSource[T](document: T)(implicit writes: Writes[T]) extends DocumentSource {
  def json: String = {
    val js = Json.toJson(document)
    Json.stringify(js)
  }
}

and indexing:

@Singleton
class VenueIndexService @Inject()(
                                  stuff...) extends ElasticIndexService[IndexableVenue] {

def indexVenue(indexableVenue: IndexableVenue) = {

    indexInto(indexableVenue, s"${indexableVenue.id.get}")
}

Why is getFields empty when doing get?
Why is query info left out when doing getSourceAsString in a get request?

Thank you!

What fields are you indexing when you add the document? There is a difference between the field and the source, and I think perhaps you are unfamiliar with the difference. — sksamuel, May 30 '14 at 15:02
I have written an index service that uses JsonSource, an extension to your DocumentSource. You are probably correct about my unfamiliarity since I'm an elastic noob =). Maybe a good response here would be that elastic4s behaves correctly, explain why and pointing to the correct documentation? If you have the time of course =D I've added the service to the question. — jakob, Jun 02 '14 at 08:19

score 3 · Accepted Answer · answered Jun 02 '14 at 08:59

What you're hitting in question 1 is that you're not specifying which fields to return. By default ES will return the source and not fields (other than type and _id). See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html

I've added a test to elastic4s to show how to retrieve fields, see: https://github.com/sksamuel/elastic4s/blob/master/src%2Ftest%2Fscala%2Fcom%2Fsksamuel%2Felastic4s%2FSearchTest.scala

I am not sure on question 2.

score 3 · Answer 2 · answered Jun 04 '14 at 11:27

The fields are empty because elasticsearch don't return it. If you need fields, you must indicate in query what field you need:

this is you search query without field:

search in "test"->"venues" query s"id:${companyId}"

and in this query we indicate which field we want to, in this case 'name' and 'description':

search in "test"->"venues" fields ("name","description") query s"id:${companyId}"

now you can retrieve the fields:

for(x <- response.getHits.hits())
      {
        println(x.getFields.get("name").getValue)

You found a getSourceAsString in a get request because the parameter _source is to default 'on' and fields is to default 'off'.

I hope this will help you

Fields are empty when doing GET in elastic4s

2 Answers2