Can Hive load data from external location which is not on HDFS?

Question

I am trying to understand that for the external table in Hive, can we have the location outside of HDFS, I mean is that I want to create my external table on top of Google storage location (gs://bucket-name/table-partitions).

score 1 · Answer 1 · edited May 23 '17 at 10:30

1

I want to update on my previously asked question that yes, it's possible with almost no code change on Google DataProc cluster or on a standalone cluster, you can follow on this link - Migrating 50TB data from local Hadoop cluster to Google Cloud Storage

edited May 23 '17 at 10:30

Community

1
1

answered Apr 11 '17 at 11:15

1

Yes, but that only partly solves your issue based on the recent changes for AWS and Google Cloud. – Aug 11 '19 at 20:37

score 0 · Accepted Answer · answered Aug 11 '19 at 20:34

0

It's not a difficult problem but requires out-of-box provisioning, something which isn't particularly well documented over Google Cloud. In order to fix it, update the following Hadoop configurations parameter:

A. Point your service account key string to that of google service account (ex: domains include @test.gservice.com) for google.cloud.auth.service.account.email.

B. Update the keyfile string to the .p12 location for google.cloud.auth.service.account.keyfile.

C. Update google.cloud.auth.service.account.enable to true.

answered Aug 11 '19 at 20:34

1

Thanks for the answer. This solution is of particular help. – Aug 11 '19 at 20:35
1

I am excited that it is of help to you. – Aug 11 '19 at 20:36

Can Hive load data from external location which is not on HDFS?

2 Answers2