Stack Overflow
Stack Overflow

Questions tagged [partitioning]

Partitioning is a performance strategy whereby you divide possibly very large groups of data into some number of smaller groups of data.

Partitioning is a performance strategy whereby you divide possibly very large groups of data into some number of smaller groups of data.

The expectation is that with algorithms of order exponentially greater than N the total time it takes to process the smaller groups and combine the results is still less than the time it would take to process the one larger set of data.

Partitioning is similar to range partitioning in many ways. As in partitioning by RANGE, each partition must be explicitly defined.

3138 questions
1
vote
2 answers

Exchange/Move partitions between tables

I have this table as my main table (100M rows): create table prova_log( id_dispositive number, type number, date_verification date, status number ) partition by range…
Marllon Nasser
  • 390
  • 4
  • 22
1
vote
1 answer

Partition rotating in MariaDB with Galera cluster

How manage partition rotating in real time on Galera Cluster? I am catch from time to time metadata lock while dropping or adding partitions on a table. I have 3 nodes in a cluster and all application connections going to first node through load…
mgavrylov
  • 11
  • 3
1
vote
2 answers

Efficient way to change the table's filegroup

I have around 300 tables which are located in different partition and now these tables are not in use for such huge data as it was. Now, I am getting space issue time to time and some of but valuable space is occupied by the 150 filegroups that was…
Shushil Bohara
  • 5,556
  • 2
  • 15
  • 32
1
vote
1 answer

partition ordering aggregateByKey Spark

So if i have a transformation before : myRDD = someRDD.map() mySecondRDD = myRDD.aggregateByKey(initValue)(CombOp , MergeOp) In this point myRDD doesn't have a partitioner, but mySecondRDD has one hashPartitioner. Firstly i want to ask: 1)Do i…
Spar
  • 463
  • 1
  • 5
  • 23
1
vote
2 answers

How to Get UTC Datetime from UNIX_TIMESTAMP() in MySQL

I want to know that how to get utc datetime from unix_timestamp in mysql. But, I should not use CONVERT_TZ. (because Could not use timezone function in partitioning.) The error occurs in the SQL schema... CREATE TABLE `table` ( `idx` BIGINT(20)…
user212942
  • 197
  • 3
  • 18
1
vote
2 answers

Hive writes to partition and DEFAULT

I have a date time table, fact table and a partitioned table. Goal is to join date_time to fact and insert in to partitioned table. It is working as I can verify /apps/hive/warehouse/dbname.db/p_tbl/p_year=2016/p_month=01/p_day=01 as well as…
AM_Hawk
  • 661
  • 1
  • 15
  • 33
1
vote
0 answers

Generate all distinct partitions of `n` balls into bins

Given 3 balls, there are 3 ways to partition them into bins: (3) (2,1) (1,1,1) Note that (1,2) is already covered in (2,1), since we don't care about the order of the bins. I'd like to write a function that gets the number of balls n and outputs…
Uri Goren
  • 13,386
  • 6
  • 58
  • 110
1
vote
1 answer

Will Postgres enforce check constraints on columns that I didn't update?

When I update a column that has a check constraint, clearly postgres has to check and see if it still meets the constraint. but if I update some other value in the row and leave all columns with check constraints unchanged, will it still do some…
Sam Schick
  • 400
  • 3
  • 14
1
vote
0 answers

How to partition a MySQL table that have PK and UNIQUE KEY?

I have this central table in my schema that is getting slow to query since it hits ~6M rows. I'm planning to alter some indexes, to this on the example, to try to better fit the queries needs. I'm also considering partitioning the table to let…
EthraZa
  • 388
  • 5
  • 9
1
vote
1 answer

Kafka - Message with different key stored in same partition

I am trying to store Messages with different key to different partition. For example: ProducerRecord rec1 = new ProducerRecord("topic", "key1", line); ProducerRecord rec2 = new ProducerRecord
Shankar
  • 8,529
  • 26
  • 90
  • 159
1
vote
1 answer

SQL Creating backup table using EXCHANGE PARTITION

I want to make a backup of table a1_crm_query using EXCHANGE PARTITION. This table contains rows with different statuses like 'ERROR', 'NEW', or 'DONE', and it would be great if there will be another table (a1_crm_query_LOG) with statuses 'ERROR'…
Jdzel
  • 159
  • 1
  • 3
  • 17
1
vote
1 answer

How to move data between partitioned tables in Hive

I have a two tables emp1 and emp2 having fields- userid name occupation country emp1 has partition on country and emp2 has partition on occupation How can I move data from emp1 to emp2
Dev
  • 13,492
  • 19
  • 81
  • 174
1
vote
2 answers

Merge Existing Partition in HIVE

How to merge existing Partitions and make it to one Partition. For ex : I have Partitions on Year column like year=2011,year=2012,year=2013,year=2014. My requirement is to merge partitions from 2011 to 2013 partitions. So that I can have only 2…
Manoj Sahoo
  • 11
  • 1
  • 4
1
vote
2 answers

how to remove the limit of partitions per node in Hive?

I have a partitioned table in Hive. And for each node I have set the partitions limit to 2000. set hive.exec.max.dynamic.partitions.pernode=2000 Now that after 2000 is reached, I am facing a problem. So i am trying to understand if there a…
user2531569
  • 609
  • 4
  • 18
  • 36
1
vote
0 answers

Is this the right way to partition data at the application level?

Currently I'm working on a legacy system that does partitioning at the application level. A line item table with a million rows (for example) have be split into two tables so that each table gets 500,000 thousand rows. I don't have experience with…
user3587180
  • 1,317
  • 11
  • 23
1 2 3
…
99 100