Kyoto TreeDB insertion slows down after 100 million records

Question

I am inserting records in Kyoto TreeDB. The key is simply a counter which I increment and the value is fixed.

After 100 million records, insertion gradually keeps slowing down. Has anyone tried to insert more records in Kyoto Cabinet?

score 1 · Accepted Answer · answered Jul 03 '12 at 19:15

1

You're probably getting hash collisions. It will put records into an overflow area if that happens. I believe that degenerates to a searched list.

answered Jul 03 '12 at 19:15

Jay

I don't see any option to tune TreeDB for higher number of records. I want to insert 1 billion records. – bazzinga Jul 03 '12 at 19:23

score 0 · Answer 2 · answered Jul 03 '12 at 19:19

0

HashDB is more memory efficient than TreeDB. Since you are using TreeDB you may want to tinker with the cache option to keep insertions fast.

answered Jul 03 '12 at 19:19

fjxx

ok. I will see into that. But I want to insert 1 billion records & in HashDB when I increase the bucket size to 2 billion, it takes a lot of time to insert. – bazzinga Jul 03 '12 at 19:22

MechanTOurS · Answer 3 · 2013-02-16T08:32:57.343

0

Did you increase bnum and msiz accordingly?

I had very poor insert performance with a too low msiz value.

edited Feb 16 '13 at 08:32

answered Feb 16 '13 at 08:24

MechanTOurS

3 Answers3