Elasticsearch: Already re-mapping but It still wont show all fields

Question

I'm just trying to sync my mongodb with ElasticSearch. I've done the sync with river, and river already worked. But the ES won't show all fields, it only show "_ts" field in the object of "_source":

Request: GET localhost:9200/test/orders/_search Response:

{
    "took": 11,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 137,
        "max_score": 1,
        "hits": [
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58a3251f761f35a107724add",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58a340467f39c50f3a54c614",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b8ec806f34179d7c7b2431",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b8eff56f3417670f7b244a",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b8f0af6f3417fb207b244c",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b8f19a6f341761337b23da",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b9320c6f3417bc1c7b23c7",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b9339f6f341777237b23c6",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b934ab6f341778237b23c7",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            },
            {
                "_index": "test",
                "_type": "orders",
                "_id": "58b96ef76f34174a4b7b23c8",
                "_score": 1,
                "_source": {
                    "_ts": 6438761296509796000
                }
            }
        ]
    }
}

I already did the remapping, and It shows the updated mapping. See this: request: GET localhost:9200/test/orders/_mapping response:

{
    "test": {
        "mappings": {
            "orders": {
                "properties": {
                    "_ts": {
                        "type": "long"
                    },
                    "activeDate": {
                        "type": "text"
                    },
                    "awbNumber": {
                        "type": "text"
                    },
                    "batchID": {
                        "type": "text"
                    },
                    "consignee": {
                        "properties": {
                            "id": {
                                "type": "text"
                            },
                            "name": {
                                "type": "text"
                            },
                            "phoneNumber": {
                                "type": "text"
                            }
                        }
                    },
                    "consigner": {
                        "properties": {
                            "id": {
                                "type": "text"
                            },
                            "name": {
                                "type": "text"
                            },
                            "phoneNumber": {
                                "type": "text"
                            }
                        }
                    },
                    "courier": {
                        "properties": {
                            "actualRate": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "id": {
                                "type": "integer"
                            },
                            "max_day": {
                                "type": "integer"
                            },
                            "min_day": {
                                "type": "integer"
                            },
                            "name": {
                                "type": "text"
                            },
                            "rate": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "rate_id": {
                                "type": "integer"
                            },
                            "rate_name": {
                                "type": "text"
                            },
                            "shipmentType": {
                                "type": "integer"
                            }
                        }
                    },
                    "creationDate": {
                        "type": "text"
                    },
                    "destination": {
                        "properties": {
                            "address": {
                                "type": "text"
                            },
                            "cityID": {
                                "type": "integer"
                            },
                            "cityName": {
                                "type": "text"
                            },
                            "id": {
                                "type": "integer"
                            },
                            "provinceID": {
                                "type": "integer"
                            },
                            "provinceName": {
                                "type": "text"
                            }
                        }
                    },
                    "driver": {
                        "properties": {
                            "feedback": {
                                "properties": {
                                    "comment": {
                                        "type": "text"
                                    },
                                    "score": {
                                        "type": "long"
                                    }
                                }
                            },
                            "id": {
                                "type": "long"
                            },
                            "isPaymentCollected": {
                                "type": "integer"
                            },
                            "name": {
                                "type": "text"
                            },
                            "phoneNumber": {
                                "type": "text"
                            },
                            "vehicleNumber": {
                                "type": "text"
                            },
                            "vehicleType": {
                                "type": "text"
                            }
                        }
                    },
                    "externalID": {
                        "type": "text"
                    },
                    "groupID": {
                        "type": "integer"
                    },
                    "id": {
                        "type": "text"
                    },
                    "isActive": {
                        "type": "long"
                    },
                    "isAutoTrack": {
                        "type": "integer"
                    },
                    "isCustomAWB": {
                        "type": "integer"
                    },
                    "isEscrow": {
                        "type": "integer"
                    },
                    "isLabelPrinted": {
                        "type": "integer"
                    },
                    "lastUpdatedDate": {
                        "type": "text"
                    },
                    "origin": {
                        "properties": {
                            "address": {
                                "type": "text"
                            },
                            "cityID": {
                                "type": "integer"
                            },
                            "cityName": {
                                "type": "text"
                            },
                            "id": {
                                "type": "integer"
                            },
                            "provinceID": {
                                "type": "integer"
                            },
                            "provinceName": {
                                "type": "text"
                            }
                        }
                    },
                    "package": {
                        "properties": {
                            "content": {
                                "type": "text"
                            },
                            "contents": {
                                "type": "integer"
                            },
                            "cubicalWeight": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "dimension": {
                                "properties": {
                                    "height": {
                                        "properties": {
                                            "UoM": {
                                                "type": "text"
                                            },
                                            "value": {
                                                "type": "integer"
                                            }
                                        }
                                    },
                                    "length": {
                                        "properties": {
                                            "UoM": {
                                                "type": "text"
                                            },
                                            "value": {
                                                "type": "integer"
                                            }
                                        }
                                    },
                                    "width": {
                                        "properties": {
                                            "UoM": {
                                                "type": "text"
                                            },
                                            "value": {
                                                "type": "integer"
                                            }
                                        }
                                    }
                                }
                            },
                            "fragile": {
                                "type": "integer"
                            },
                            "isConfirmed": {
                                "type": "integer"
                            },
                            "itemName": {
                                "type": "text"
                            },
                            "itemSubtype": {
                                "type": "integer"
                            },
                            "itemType": {
                                "type": "integer"
                            },
                            "pictureURL": {
                                "type": "text"
                            },
                            "price": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "type": {
                                "type": "integer"
                            },
                            "weight": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            }
                        }
                    },
                    "paymentType": {
                        "type": "text"
                    },
                    "pickUpTime": {
                        "type": "text"
                    },
                    "rates": {
                        "properties": {
                            "actualInsurance": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "long"
                                    }
                                }
                            },
                            "actualShipment": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "escrowCost": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "fulfillmentCost": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "insurance": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "long"
                                    }
                                }
                            },
                            "itemPrice": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "liability": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            },
                            "shipment": {
                                "properties": {
                                    "UoM": {
                                        "type": "text"
                                    },
                                    "value": {
                                        "type": "integer"
                                    }
                                }
                            }
                        }
                    },
                    "readyTime": {
                        "type": "text"
                    },
                    "shipmentStatus": {
                        "properties": {
                            "description": {
                                "type": "text"
                            },
                            "name": {
                                "type": "text"
                            },
                            "statusCode": {
                                "type": "integer"
                            },
                            "updateDate": {
                                "type": "text"
                            },
                            "updatedBy": {
                                "type": "text"
                            }
                        }
                    },
                    "source": {
                        "type": "text"
                    },
                    "specialID": {
                        "type": "text"
                    },
                    "stickerNumber": {
                        "type": "text"
                    },
                    "useInsurance": {
                        "type": "integer"
                    }
                }
            }
        }
    }
}

I'm expecting I can get the whole fields (not just a single "_ts" field) in the "_source" field which it should be the same with what I've mapped.

I've tried to delete the index and recreate one, still not worked. Any clue of this kind of issue. I really really need help, thank you so much.

score 0 · Accepted Answer · edited Jul 04 '17 at 05:56

0

ElasticSearch has a behavior of creating fields on its own. So if you do

PUT document/index/1
{
    "id" : "1",
    "name" : "kashish",
    "phoneNumber" : "9740683281"
}

This will automatically create fields for your index. What I am illustrating is if explicitly defining of fields is not working for you for some reason , you can empty the index(if contains dummy data) and then just put your json which ES will pickup automatically.

edited Jul 04 '17 at 05:56

Roopendra

7,674
16
65
92

answered Jul 04 '17 at 04:49

Kashish Verma

26
2

Do you mean by inserting one document right after creating the index? I've tried it but still not worked. – Thomson Palito Napitupulu Jul 04 '17 at 05:06
Do not create a document or index. Just put request without creating any document and index. It will automatically pick the given json and based on fields name it will configure field types for you. Hope it helps. – Kashish Verma Jul 04 '17 at 05:46
Sorry but I don't catch it. I didn't create the document on ES, I created it on mongodb shell and it automatically sync into ES. I don't know how to not create index but it automatically created. Could you give me some example on how to create index automatically. Thank youu @Kashish Verma – Thomson Palito Napitupulu Jul 04 '17 at 05:53
Could you give me a simple example of that put request? I'm still newbie on ES, hard to get it at a glance. Thank youu – Thomson Palito Napitupulu Jul 04 '17 at 06:02
1

I've tried the put request like on your first answer, and yes, it's show all fields. But actually It still could not solve my issue. After I did the put request, I tried to copy a collection from another db to my current db and after I finished copying, they still didn't show all the fields. Do you have any idea about copying a collection but still same like put request? – Thomson Palito Napitupulu Jul 04 '17 at 06:27
I want to do "db.copyDatabase()" function instead of put request, did you have any idea on how to get the same result like put request but by doing db.copyDatabase() on mongo shell? Thanks for your kind response. – Thomson Palito Napitupulu Jul 04 '17 at 06:33
And also, the mongo and ES is not two-way sync. The data I just stored by doing put request in ES, is not exist in my mongodb. the mongo itself synced into ES, but the ES didn't sync into mongo. (sorry for my bad explanation). So, put request worked but is not good to solve my issue. Thanks :) – Thomson Palito Napitupulu Jul 04 '17 at 06:48
It Depends when are you synchronizing both DB and ES. After u have all your database rows then you can go ahead and sync in order to ensure that both DB and ES are synchronized. When you are inserting a row in database you can write logic there itself for deleting previous ES data and copy/paste all data from db into ES again. But again if you directly make changes in ES through put request , the new data would not be reflected in your DB. My Suggestions would be write an external API which copy/paste data from DB to ES , so that you can call it anytime to make data consistency in DB and ES. – Kashish Verma Jul 04 '17 at 09:42
I think you're right. The problem is not in the ES but in the river. Anyway your suggestion is also acceptable. I'll try it for my alternative solution. Thanks @Kashish Verma – Thomson Palito Napitupulu Jul 04 '17 at 13:37

Elasticsearch: Already re-mapping but It still wont show all fields

1 Answers1