Presto and hive partition discovery

Question

I'm using presto mainly with hive connector to connect to hive metastore.

All of my tables are external tables pointing to data stored in S3.

My main issue with this is that there is no way (at least on I'm aware of ) to do partition discovery in Presto ,so before I start query a table in presto I need to switch to hive and run msck repair table mytable

is there more reasonable way to do it in Presto?

score 5 · Answer 1 · answered May 12 '20 at 15:37

5

I'm on version 0.227 and the following helps me:

select * from hive.yourschema."yourtable$partitions"

This select returns all the partitions mapped in your catalog. You can filter, order, etc. as a normal query would.

answered May 12 '20 at 15:37

Ricardo Pedrotti

83
1
7

score 3 · Accepted Answer · answered Dec 30 '15 at 14:10

3

No.

If the HIVE metastore doesn't see the partitions, PrestoDB will not see it.

Maybe a cron can help you.

answered Dec 30 '15 at 14:10

Damien Carol

1,164
1
11
21

2

Also, it looks like there is a cache between PrestoDB & Hive partition, I don't find anything about that but I always have to wait 2 minutes before running queries on Presto. – Thomas Decaux May 04 '17 at 13:29

score 0 · Answer 3 · answered Jul 22 '21 at 02:12

0

There is now a way to do this:

CALL system.sync_partition_metadata(schema_name=>'<your-schema>', table_name=>'<your-table>', mode=>'FULL')

Credit to this post and this video

answered Jul 22 '21 at 02:12

Lou Zell

5,255
3
28
23

Presto and hive partition discovery

3 Answers3