0

I was wondering if it's possible to do partitioning after table creation.

I'm trying to import ~2 million entries in a table (cluster), and if I partition the table before adding entries I get memory exceptions.

gtonic
  • 2,295
  • 1
  • 24
  • 32
Helge Waastad
  • 151
  • 1
  • 11
  • maybe your import creates a lot of partitions. what is your partition value? – Christian Sep 11 '15 at 06:34
  • maybe your configured heap is to low. whats your configured heap on each instance? You can find out in the admin-ui or by issueing: `select id, name, (mem['free'] + mem['used'])/1024/1024 as configured_heap_mb, mem['free']/1024/1024 as free_heap_mb from sys.nodes;` – mwahl Sep 11 '15 at 07:05
  • Hi,this actually initial testing in migrating an postgres installation to Crate. It's typical a timeseries type so I've made a partition by day (~2 years collections) so I guess number of partitions gets high. – Helge Waastad Sep 11 '15 at 08:00
  • yeah, you'll get a lot of partitions, but more importantly a lot of shards! `2 * 365 * num_shards` so if you have your partitioned table with 4 shards, you will get 2920 shards in total. try to partition by month, or at least by week to minimize the number of partitions. – Christian Sep 11 '15 at 09:55

1 Answers1

1

It is not possible to partition a table after its creation.

2M records is not a lot of data, but if you have problems importing data into a partitioned table (e.g. if you have a lot of partitions), you could import data per partition:

COPY table_name PARTITION (partition_column=column_name) WITH (option=value);

See COPY FROM.

Christian
  • 279
  • 2
  • 8