Combine Multiple Hive Tables as single table in Hadoop

Question

Hi I have multiple Hive tables around 15-20 tables. All the tables will be common schema . I Need to combine all the tables as single table.The single table should be queried from reporting tool, So performance is also needs to be care..

I tried like this..

create table new as
select * from table_a
union all
select * from table_b

Is there any other way to combine all the tables more efficient. Any help will be appreciated.

score 0 · Answer 1 · answered Jul 26 '18 at 07:27

0

Hive would be processing in parallel if you set "hive.exec.parallel" as true. With "hive.exec.parallel.thread.number" you can specify the number of parallel threads. This would increase the overall efficiency.

answered Jul 26 '18 at 07:27

Debottam Rakshit

31
4

score 0 · Answer 2 · answered Jul 26 '18 at 07:30

0

If you are trying to merge table_A and table_b into a single one, the easiest way is to use the UNION ALL operator. You can find the syntax and use cases here - https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Union

answered Jul 26 '18 at 07:30

Gautham Srinivasan

308
3
19

But unionall is costly operation right for multiple tables?? – Teju Priya Jul 26 '18 at 08:13
could you use MAPJOIN in that instead of Unionall – Gautham Srinivasan Jul 26 '18 at 08:24

Combine Multiple Hive Tables as single table in Hadoop

2 Answers2