1

I'm trying to load a CSV from 1 remote server to a Hive client on a different server using Python:

I'm opening the CSV file on remote server:

with open("/path/to/csv/file/" +self.file_to_load, "rb") as file:
    csv_file = file.read()

Now i'm looking for the command to load that csv_file into Hive (which is on a different server) using pyhive connect;

conn_h = hive.connect(host=self.hive_host, port=10000, auth='NONE',username=self.user)
    
query = "LOAD LOCAL DATA INPATH 'file://" + self.path_to_file + "' OVERWRITE INTO TABLE " + self.tgt_hive_table
conn_h.cursor.execute(query)

I'm looking for a way to load csv file without saving the actual file on server

martineau
  • 119,623
  • 25
  • 170
  • 301
Arik
  • 55
  • 7
  • i think you actually will need to upload the file to the HDFS for this to work, any reason you don't want to do this? – Cyclonecode Aug 19 '20 at 13:35
  • That's what i ended up doing @martineau, I created an external table on the csv file. The reason was security issues. – Arik Jan 04 '21 at 07:15
  • I think you should have addressed your comment to @Cyclonecode, not me. – martineau Jan 04 '21 at 07:36

0 Answers0