0

After indexing ddb records into ElasticSearch, when doing a simple search /_search?q=test, I see the hits shown like this

"hits": [
            {
                // ignore other fields ...
                "_id": "z0YdS3I",
                "_source": {
                    "M": {
                        "name": {
                            "S": "test name"
                        },
                        "age": {
                            "N": "18"
                        },
                        // ignore other fields ...
                    }
                }
            },
            ....
        ]

However, when I search for a specific field, e.g. /_search?q=name:test, I get zero hits. This happens with every field. 

"hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }

So instead I have to search like this _search?q=M.name.S=test, which is a bit cumbersome. Just wonder if there's a cleaner way to search for a field? Maybe I'm missing some configuration during indexing step?

chepukha
  • 2,371
  • 3
  • 28
  • 40
  • the "test" text is inside `M.name.S` and not just under field `name` . why is that ? is that what you desire ? or some problem while indexing the data ? – Sowjanya R Bhat May 25 '20 at 20:38
  • It was not what I intended. When adding data to ES, if I add a normal object, I'll get 400 "'object mapping for [name] tried to parse field [name] as object, but found a concrete value'". ES was expecting input in that format. So instead I have to search for M.name.S which a bit cumbersome. – chepukha May 25 '20 at 20:54
  • so you could try to first define mappings for your index as per your requirement . like - "name":"text", "age":ïnteger" etc . then check if that got applied properly using '/_mapping' API - once you see the datatypes are proper then start indexing data into Elasticsearch. – Sowjanya R Bhat May 25 '20 at 21:00
  • ah, I see. that could be the reason. I didn't define mapping before adding data to ES. But one problem is that since DDB table can change, it's hard to define the mapping in advance. But I think you're right. Feel free to post the answer, so I can accept it. Thanks – chepukha May 25 '20 at 21:02

2 Answers2

1

You could try this :

First define mappings for your index as per your requirement . like - 

"name":"text", 
"age":"integer"
.
.
etc

Then check if that got applied properly using /_mapping API - once you see the datatypes are applied as you desire then start indexing data.

Details of mappings => https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

Sowjanya R Bhat
  • 1,128
  • 10
  • 19
  • 1
    +1 Thanks @sowjany-r-hhat, but since the ddb documents can change (attributes can be added and be missing), this approach is not suitable in my case. I also found out AWS provided a library to convert DDB document to normal JS object. I posted an answer with details. – chepukha May 26 '20 at 23:44
  • @chepukha so you are syncing Dynamo data to ES . cool ! so in future if i have the same usecase of pulling data from dynamo n loading it to ES i should be using `dynamo-converter` . my current case is i dump data to ES using python - where i first define mapping - then index it. you might find this link also useful (for general ES knowledge) => https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html – Sowjanya R Bhat May 27 '20 at 07:03
1

I found out I could use DynamoDB Converter provided by AWS SDK to convert back and forth between Javascript object and its equivalent DDB AttributeValue type. That way I can index a document in the write mapping and access it with the normal fields.

chepukha
  • 2,371
  • 3
  • 28
  • 40