Stack Overflow
Stack Overflow

Questions tagged [hive-partitions]

To be used for questions regarding partitions in hive.

Partitioning is a way of dividing a table into related parts based on the values of partitioned columns such as date, city, and department. Using partition, it is easy to query a portion of the data.

Partitions are essentially horizontal slices of data which allow larger sets of data to be separated into more manageable chunks. In Hive, partitioning is supported for both managed and external tables in the table definition as seen below.

144 questions
1
vote
1 answer

Insert data in many partitions using one insert statement

I have table A and table B, where B is the partitioned table of A using a field called X. When I want to insert data from A to B, I usually execute the following statement: INSERT INTO TABLE B PARTITION(X=x) SELECT FROM A WHERE…
JaviOverflow
  • 1,434
  • 2
  • 14
  • 31
1
vote
0 answers

Hive describe shows partition also as column but describe formatted doesn't

Hive table created: create external table ini(id string, rand string) partitioned by (tmp string) Describe: describe ini; Output from hue: Describe formatted: describe formatted ini; Output from hue: Why is the partition column shown in column…
Ani Menon
  • 27,209
  • 16
  • 105
  • 126
1
vote
2 answers

Delete partition directories from HDFS, would it reflect in hive table?

Lets say I created a hive table with partition column as year, month and day and if i delete the partition from hdfs, then result get reflected in hive table or not
Dipak
  • 39
  • 2
  • 9
1
vote
1 answer

Hive - static partitioning - difference between creating the partition directory directly vs using alter table statement

Are there any internal/performance difference between the below two statements for creating static partitioning in hive, I have tried both ways and both of them are working without any issues after loading the data into partition dfs -mkdir…
venkata
  • 447
  • 3
  • 15
1
vote
1 answer

error when inserting data to a partitioned table in Hive

I am seeing an error when I try to insert data into a partitioned table in hive, here is the details: The table: CREATE　TABLE partitionedemp(emp_no int, birth_date string, first_name string, last_name string) Partitioned By(gender string, hire_date…
mdivk
  • 3,545
  • 8
  • 53
  • 91
1
vote
1 answer

Nesting Static Partition under Dynamic Partition

In Hive why I am not allowed to nest a static partition under a dynamic partition? for eg the below is allowed INSERT OVERWRITE TABLE T PARTITION (ds='2010-03-03', hr) SELECT key, value, /*ds,*/ hr FROM srcpart WHERE ds is not null and hr>10; but…
rogue-one
  • 11,259
  • 7
  • 53
  • 75
0
votes
0 answers

BigQuery external table over GCS path with partitions

I have some data stored in GCS bucket in the following path: gcs://my-bucket/my_data/subfolder1/subfolder2/**.csv.gz I intent to create an external table mapping to my_data and want the external table is able to partition the data by different…
Lee
  • 2,874
  • 3
  • 27
  • 51
0
votes
0 answers

Trino Invalid partition spec:

I've created a external partition table in trino, I'm using hive connector. I'm changing the partition location with unregister partition using command system.unregister_partition(schema_name, table_name, partition_columns, partition_values) and…
Nikhil Lingam
  • 121
  • 2
  • 12
0
votes
1 answer

Need to merge multiple hive partitions into one partition in spark

I have around 50 partitions in hive table. I need to merge each set of partitions into one partition. I tried to use rename partition command. But getting error message. Need help in merging multiple hive partitions into one partition in spark …
Arvinth
  • 60
  • 6
  • 27
0
votes
0 answers

HIVE: Exception: Partition Already Exists while ADDING a NEW Partition to an EXISTING EXTERNAL Table

I am getting the below error when the application (java) tries to execute an 'ADD partition' after 'DROP partition IF EXISTS' command in Hive:- """ Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return…
Priyabrata Behera
  • 1
  • 1
0
votes
1 answer

How to create partitions (year,month,day) in hive from date column which have MM/dd/yyyy format

Data loaded on a daily basis. Need to create a partition with the date column. Date 3/15/2021 8:02:32 AM 12/21/2020 12:20:41 PM
Jaffar Ahamed
  • 21
  • 3
0
votes
0 answers

How to drop partitions from hive views?

I have a partitioned view and I am trying to drop an existing partition from the view definition using hive CLI. However, when I try to drop a partition, it throws me the following error: FAILED: Execution Error, return code 1 from…
Rishi Pathak
  • 1
0
votes
1 answer

Missing hive partition key column while creating hive partition external table using bq command

I am using the bq version 2.0.58 and trying to create hive partitioning BigQuery external table. But it's missing hive partition key column while creating hive partition external table using bq command, the following command to create a…
SST
  • 2,054
  • 5
  • 35
  • 65
0
votes
1 answer

Hive sql - between multiple partitions

I got a hive table partitioned by year, month and day CREATE TABLE t1 ( ... ) PARTITIONED BY ( year INT, month INT, day INT' ) STORED AS PARQUET; and I need to take the data from it last 7 days/partitions. The first step towards finding how to…
fqv572
  • 39
  • 5
0
votes
1 answer

Error while updating records on database partitions using Spring JPA and Hibernate

I am getting exception on update of Hibernate entity, below are the details - I have created tables partitions as below --case --case_active --case_inactive ---------------Script------------ CREATE TABLE user_active ( …
Jitendra Rathore
  • 1
  • 1
1 2 3
…
9 10