0

jaydebeapi executemany() method is not working for big csv file writing to hadoop table.

Can someone please give example to writing csv data to Hive table?

OneManFunction
  • 31
  • 2
  • 5
  • "Hadoop table" is not a thing. Did you mean Hive? You should use PySpark for this, not a JDBC Driver – OneCricketeer Feb 14 '22 at 18:31
  • @OneCricketeer I am new here so i was not aware about hive table ..can you please suggest how to insert into hive table through pyspark and why jaydebeapi is not good option for this problem – OneManFunction Feb 15 '22 at 05:19
  • Hadoop itself has nothing related to JDBC, so what JDBC connection are you trying to make? – OneCricketeer Feb 15 '22 at 13:50

1 Answers1

0

big csv file writing to hadoop

Unclear why you're trying to use JDBC for this

pip install pyspark

import pyspark 

from pyspark.sql import SparkSession 

spark = SparkSession.getOrCreate()
df = spark.read.csv("file.csv")
df.write.parquet("hdfs:///tmp/upload")

Alternatively, if you're using Apache Hive, then see https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245