Count mismatch in Hive- Elasticsearch integration in hive table and ES index

Question

I had created Elastic search index using hive. Here, I have one temp table, where load all the raw data. From that table select some data on some criteria and insert them to a table which is integrated with Elastic search index.

After index creation I am comparing the count at hive table (in the main table on same criteria), on the table integrated with ES and elastic search index. found count does not same.

In ES index it is: 4663296 On table integrated with ES: 4663296 (same as ES) but in hive it's : 4611296 (main table on same criteria) - less then ES

So could some one please tell me why this count is more in ES. It should be same, am I right?

Thanks, Rackto

Hi Pavan, I was facing the issue due to duplicate records. I have added unique key at table properties. Please check the below answer. Thanks — user1321939, Feb 17 '17 at 07:42

score 0 · Answer 1 · answered Aug 06 '14 at 13:13

0

It was found that there was some duplicate records in the ES.

So, what I am doing, add the id manually (some key in the data which is always unique), now the count is same.

Just need to add one table properties: TBLPROPERTIES('......., 'es.mapping.id' = 'field_name_of_the_unique_id'); in hive table creation.

Thanks

answered Aug 06 '14 at 13:13

user1321939

319
2
6
18

what if we dont have any unique field in our document? – Pavan Kumar Varma Feb 17 '17 at 09:05
I think, you can create combine_key to generate one unique key while creating the ES tables in hive. You create it by using "concat" function available at hive. – user1321939 Feb 20 '17 at 09:53

Count mismatch in Hive- Elasticsearch integration in hive table and ES index

1 Answers1