I have 1500 partition in my hive tables but while doing query it is taking more time then expected. Maximum number of partitions can be created in hive table.
Asked
Active
Viewed 6,864 times
2
-
Please give more description of your question. like sample code, etc. – E.Coms Oct 22 '18 at 13:13
1 Answers
3
Current Hive versions with RDBMS metastore backend should be able to handle 10000+ partitions.
For numerous reasons, the community is moving away from this design to leverage HBase for the metastore. Follow https://issues.apache.org/jira/browse/HIVE-9452 .
Overall design document is available here: https://issues.apache.org/jira/secure/attachment/12697601/HBaseMetastoreApproach.pdf
Mayank Porwal
- 33,470
- 8
- 37
- 58
-
Go through this: https://stackoverflow.com/questions/29103221/partitioning-in-hive – Mayank Porwal Oct 22 '18 at 13:01