Kyoto Cabinet HashDB inserts are not O(1)

Question

I am experimenting with Kyoto Cabinet HashDB. My database will grow up to the size of 110 TB. The keys are 20 Bytes and values are 128 KB. I am trying to insert about 500 million records. Here is the best performance I have gotten so far:

Time is in seconds. 'n' is the number of 10000 records entered. For example 1st 10000 records took 1 second. 5th 10000 records took 5 seconds and so on. As you can see the performance starts degrading after about 2 million records are entered. I created the database with bucket number set to 7 Billion, and everything else is default. I am using the C/C++ library, and setting memory map capacity to 30 GB (I have 32 GB RAM) using db.tune_map(30LL << 30);.

Inserted 10000 records. n: 1. Time: 1
Inserted 10000 records. n: 2. Time: 1
Inserted 10000 records. n: 3. Time: 1
Inserted 10000 records. n: 4. Time: 1
Inserted 10000 records. n: 5. Time: 4
Inserted 10000 records. n: 6. Time: 4
Inserted 10000 records. n: 7. Time: 4
Inserted 10000 records. n: 8. Time: 4
Inserted 10000 records. n: 9. Time: 4
Inserted 10000 records. n: 10. Time: 4
Inserted 10000 records. n: 11. Time: 4
Inserted 10000 records. n: 12. Time: 4
Inserted 10000 records. n: 13. Time: 4
Inserted 10000 records. n: 14. Time: 4
Inserted 10000 records. n: 15. Time: 4
Inserted 10000 records. n: 16. Time: 4
Inserted 10000 records. n: 17. Time: 4
Inserted 10000 records. n: 18. Time: 4
Inserted 10000 records. n: 19. Time: 7
Inserted 10000 records. n: 20. Time: 4
Inserted 10000 records. n: 21. Time: 4
Inserted 10000 records. n: 22. Time: 4
Inserted 10000 records. n: 23. Time: 5
Inserted 10000 records. n: 24. Time: 4
Inserted 10000 records. n: 25. Time: 4
Inserted 10000 records. n: 26. Time: 4
Inserted 10000 records. n: 27. Time: 4
Inserted 10000 records. n: 28. Time: 4
Inserted 10000 records. n: 29. Time: 5
Inserted 10000 records. n: 30. Time: 4
Inserted 10000 records. n: 31. Time: 4
Inserted 10000 records. n: 32. Time: 4
Inserted 10000 records. n: 33. Time: 4
Inserted 10000 records. n: 34. Time: 4
Inserted 10000 records. n: 35. Time: 4
Inserted 10000 records. n: 36. Time: 4
Inserted 10000 records. n: 37. Time: 5
Inserted 10000 records. n: 38. Time: 4
Inserted 10000 records. n: 39. Time: 4
Inserted 10000 records. n: 40. Time: 4
Inserted 10000 records. n: 41. Time: 4
Inserted 10000 records. n: 42. Time: 4
Inserted 10000 records. n: 43. Time: 4
Inserted 10000 records. n: 44. Time: 4
Inserted 10000 records. n: 45. Time: 4
Inserted 10000 records. n: 46. Time: 4
Inserted 10000 records. n: 47. Time: 4
Inserted 10000 records. n: 48. Time: 5
Inserted 10000 records. n: 49. Time: 4
Inserted 10000 records. n: 50. Time: 4
Inserted 10000 records. n: 51. Time: 5
Inserted 10000 records. n: 52. Time: 4
Inserted 10000 records. n: 53. Time: 5
Inserted 10000 records. n: 54. Time: 5
Inserted 10000 records. n: 55. Time: 4
Inserted 10000 records. n: 56. Time: 5
Inserted 10000 records. n: 57. Time: 5
Inserted 10000 records. n: 58. Time: 5
Inserted 10000 records. n: 59. Time: 5
Inserted 10000 records. n: 60. Time: 5
Inserted 10000 records. n: 61. Time: 4
Inserted 10000 records. n: 62. Time: 5
Inserted 10000 records. n: 63. Time: 5
Inserted 10000 records. n: 64. Time: 4
Inserted 10000 records. n: 65. Time: 6
Inserted 10000 records. n: 66. Time: 4
Inserted 10000 records. n: 67. Time: 4
Inserted 10000 records. n: 68. Time: 7
Inserted 10000 records. n: 69. Time: 6
Inserted 10000 records. n: 70. Time: 7
Inserted 10000 records. n: 71. Time: 7
Inserted 10000 records. n: 72. Time: 6
Inserted 10000 records. n: 73. Time: 10
Inserted 10000 records. n: 74. Time: 7
Inserted 10000 records. n: 75. Time: 10
Inserted 10000 records. n: 76. Time: 9
Inserted 10000 records. n: 77. Time: 8
Inserted 10000 records. n: 78. Time: 10
Inserted 10000 records. n: 79. Time: 8
Inserted 10000 records. n: 80. Time: 6
Inserted 10000 records. n: 81. Time: 8
Inserted 10000 records. n: 82. Time: 8
Inserted 10000 records. n: 83. Time: 9
Inserted 10000 records. n: 84. Time: 9
Inserted 10000 records. n: 85. Time: 9
Inserted 10000 records. n: 86. Time: 9
Inserted 10000 records. n: 87. Time: 7
Inserted 10000 records. n: 88. Time: 10
Inserted 10000 records. n: 89. Time: 8
Inserted 10000 records. n: 90. Time: 10
Inserted 10000 records. n: 91. Time: 10
Inserted 10000 records. n: 92. Time: 10
Inserted 10000 records. n: 93. Time: 10
Inserted 10000 records. n: 94. Time: 10
Inserted 10000 records. n: 95. Time: 10
Inserted 10000 records. n: 96. Time: 11
Inserted 10000 records. n: 97. Time: 9
Inserted 10000 records. n: 98. Time: 10
Inserted 10000 records. n: 99. Time: 10
Inserted 10000 records. n: 100. Time: 11
Inserted 10000 records. n: 101. Time: 10
Inserted 10000 records. n: 102. Time: 11
Inserted 10000 records. n: 103. Time: 11
Inserted 10000 records. n: 104. Time: 11
Inserted 10000 records. n: 105. Time: 12
Inserted 10000 records. n: 106. Time: 11
Inserted 10000 records. n: 107. Time: 12
Inserted 10000 records. n: 108. Time: 12
Inserted 10000 records. n: 109. Time: 10
Inserted 10000 records. n: 110. Time: 11
Inserted 10000 records. n: 111. Time: 10
Inserted 10000 records. n: 112. Time: 10
Inserted 10000 records. n: 113. Time: 11
Inserted 10000 records. n: 114. Time: 10
Inserted 10000 records. n: 115. Time: 11
Inserted 10000 records. n: 116. Time: 10
Inserted 10000 records. n: 117. Time: 10
Inserted 10000 records. n: 118. Time: 10
Inserted 10000 records. n: 119. Time: 10
Inserted 10000 records. n: 120. Time: 11
Inserted 10000 records. n: 121. Time: 10
Inserted 10000 records. n: 122. Time: 11
Inserted 10000 records. n: 123. Time: 9
Inserted 10000 records. n: 124. Time: 11
Inserted 10000 records. n: 125. Time: 9
Inserted 10000 records. n: 126. Time: 11
Inserted 10000 records. n: 127. Time: 10
Inserted 10000 records. n: 128. Time: 10
Inserted 10000 records. n: 129. Time: 12
Inserted 10000 records. n: 130. Time: 10
Inserted 10000 records. n: 131. Time: 10
Inserted 10000 records. n: 132. Time: 10
Inserted 10000 records. n: 133. Time: 10
Inserted 10000 records. n: 134. Time: 16
Inserted 10000 records. n: 135. Time: 10
Inserted 10000 records. n: 136. Time: 10
Inserted 10000 records. n: 137. Time: 10
Inserted 10000 records. n: 138. Time: 10
Inserted 10000 records. n: 139. Time: 9
Inserted 10000 records. n: 140. Time: 10
Inserted 10000 records. n: 141. Time: 10
Inserted 10000 records. n: 142. Time: 10
Inserted 10000 records. n: 143. Time: 10
Inserted 10000 records. n: 144. Time: 11
Inserted 10000 records. n: 145. Time: 10
Inserted 10000 records. n: 146. Time: 11
Inserted 10000 records. n: 147. Time: 9
Inserted 10000 records. n: 148. Time: 10
Inserted 10000 records. n: 149. Time: 9
Inserted 10000 records. n: 150. Time: 10
Inserted 10000 records. n: 151. Time: 20
Inserted 10000 records. n: 152. Time: 18
Inserted 10000 records. n: 153. Time: 33
Inserted 10000 records. n: 154. Time: 25
Inserted 10000 records. n: 155. Time: 30
Inserted 10000 records. n: 156. Time: 18
Inserted 10000 records. n: 157. Time: 12
Inserted 10000 records. n: 158. Time: 13
Inserted 10000 records. n: 159. Time: 14
Inserted 10000 records. n: 160. Time: 10
Inserted 10000 records. n: 161. Time: 9
Inserted 10000 records. n: 162. Time: 12
Inserted 10000 records. n: 163. Time: 11
Inserted 10000 records. n: 164. Time: 11
Inserted 10000 records. n: 165. Time: 11
Inserted 10000 records. n: 166. Time: 11
Inserted 10000 records. n: 167. Time: 12
Inserted 10000 records. n: 168. Time: 10
Inserted 10000 records. n: 169. Time: 11
Inserted 10000 records. n: 170. Time: 12
Inserted 10000 records. n: 171. Time: 11
Inserted 10000 records. n: 172. Time: 10
Inserted 10000 records. n: 173. Time: 11
Inserted 10000 records. n: 174. Time: 11
Inserted 10000 records. n: 175. Time: 10
Inserted 10000 records. n: 176. Time: 11
Inserted 10000 records. n: 177. Time: 11
Inserted 10000 records. n: 178. Time: 10
Inserted 10000 records. n: 179. Time: 11
Inserted 10000 records. n: 180. Time: 11
Inserted 10000 records. n: 181. Time: 11
Inserted 10000 records. n: 182. Time: 10
Inserted 10000 records. n: 183. Time: 11
Inserted 10000 records. n: 184. Time: 11
Inserted 10000 records. n: 185. Time: 11
Inserted 10000 records. n: 186. Time: 11
Inserted 10000 records. n: 187. Time: 11
Inserted 10000 records. n: 188. Time: 11
Inserted 10000 records. n: 189. Time: 11
Inserted 10000 records. n: 190. Time: 11
Inserted 10000 records. n: 191. Time: 11
Inserted 10000 records. n: 192. Time: 13
Inserted 10000 records. n: 193. Time: 13
Inserted 10000 records. n: 194. Time: 13
Inserted 10000 records. n: 195. Time: 12
Inserted 10000 records. n: 196. Time: 13
Inserted 10000 records. n: 197. Time: 11
Inserted 10000 records. n: 198. Time: 11
Inserted 10000 records. n: 199. Time: 12
Inserted 10000 records. n: 200. Time: 11
Inserted 10000 records. n: 201. Time: 11
Inserted 10000 records. n: 202. Time: 11
Inserted 10000 records. n: 203. Time: 11
Inserted 10000 records. n: 204. Time: 11
Inserted 10000 records. n: 205. Time: 12
Inserted 10000 records. n: 206. Time: 11
Inserted 10000 records. n: 207. Time: 10
Inserted 10000 records. n: 208. Time: 10
Inserted 10000 records. n: 209. Time: 11
Inserted 10000 records. n: 210. Time: 10
Inserted 10000 records. n: 211. Time: 10
Inserted 10000 records. n: 212. Time: 11
Inserted 10000 records. n: 213. Time: 11
Inserted 10000 records. n: 214. Time: 10
Inserted 10000 records. n: 215. Time: 11
Inserted 10000 records. n: 216. Time: 11
Inserted 10000 records. n: 217. Time: 12
Inserted 10000 records. n: 218. Time: 12
Inserted 10000 records. n: 219. Time: 12
Inserted 10000 records. n: 220. Time: 11
Inserted 10000 records. n: 221. Time: 12
Inserted 10000 records. n: 222. Time: 10
Inserted 10000 records. n: 223. Time: 11
Inserted 10000 records. n: 224. Time: 11
Inserted 10000 records. n: 225. Time: 11
Inserted 10000 records. n: 226. Time: 11
Inserted 10000 records. n: 227. Time: 11
Inserted 10000 records. n: 228. Time: 11
Inserted 10000 records. n: 229. Time: 11
Inserted 10000 records. n: 230. Time: 11
Inserted 10000 records. n: 231. Time: 10
Inserted 10000 records. n: 232. Time: 11
Inserted 10000 records. n: 233. Time: 10
Inserted 10000 records. n: 234. Time: 11
Inserted 10000 records. n: 235. Time: 11
Inserted 10000 records. n: 236. Time: 11
Inserted 10000 records. n: 237. Time: 11
Inserted 10000 records. n: 238. Time: 10
Inserted 10000 records. n: 239. Time: 11
Inserted 10000 records. n: 240. Time: 11
Inserted 10000 records. n: 241. Time: 11
Inserted 10000 records. n: 242. Time: 11
Inserted 10000 records. n: 243. Time: 11
Inserted 10000 records. n: 244. Time: 11
Inserted 10000 records. n: 245. Time: 11
Inserted 10000 records. n: 246. Time: 12
Inserted 10000 records. n: 247. Time: 11
Inserted 10000 records. n: 248. Time: 12
Inserted 10000 records. n: 249. Time: 12
Inserted 10000 records. n: 250. Time: 11
Inserted 10000 records. n: 251. Time: 12
Inserted 10000 records. n: 252. Time: 12
Inserted 10000 records. n: 253. Time: 13
Inserted 10000 records. n: 254. Time: 10
Inserted 10000 records. n: 255. Time: 12
Inserted 10000 records. n: 256. Time: 12
Inserted 10000 records. n: 257. Time: 13
Inserted 10000 records. n: 258. Time: 11
Inserted 10000 records. n: 259. Time: 10
Inserted 10000 records. n: 260. Time: 12
Inserted 10000 records. n: 261. Time: 11
Inserted 10000 records. n: 262. Time: 11
Inserted 10000 records. n: 263. Time: 12
Inserted 10000 records. n: 264. Time: 10
Inserted 10000 records. n: 265. Time: 10
Inserted 10000 records. n: 266. Time: 11
Inserted 10000 records. n: 267. Time: 10
Inserted 10000 records. n: 268. Time: 11
Inserted 10000 records. n: 269. Time: 12
Inserted 10000 records. n: 270. Time: 12
Inserted 10000 records. n: 271. Time: 11
Inserted 10000 records. n: 272. Time: 11
Inserted 10000 records. n: 273. Time: 10
Inserted 10000 records. n: 274. Time: 12
Inserted 10000 records. n: 275. Time: 9
Inserted 10000 records. n: 276. Time: 12
Inserted 10000 records. n: 277. Time: 11
Inserted 10000 records. n: 278. Time: 11
Inserted 10000 records. n: 279. Time: 11
Inserted 10000 records. n: 280. Time: 10
Inserted 10000 records. n: 281. Time: 11
Inserted 10000 records. n: 282. Time: 11
Inserted 10000 records. n: 283. Time: 10
Inserted 10000 records. n: 284. Time: 10
Inserted 10000 records. n: 285. Time: 11
Inserted 10000 records. n: 286. Time: 10
Inserted 10000 records. n: 287. Time: 10
Inserted 10000 records. n: 288. Time: 11
Inserted 10000 records. n: 289. Time: 10
Inserted 10000 records. n: 290. Time: 9
Inserted 10000 records. n: 291. Time: 16
Inserted 10000 records. n: 292. Time: 10
Inserted 10000 records. n: 293. Time: 9
Inserted 10000 records. n: 294. Time: 11
Inserted 10000 records. n: 295. Time: 9
Inserted 10000 records. n: 296. Time: 9
Inserted 10000 records. n: 297. Time: 11
Inserted 10000 records. n: 298. Time: 39
Inserted 10000 records. n: 299. Time: 52
Inserted 10000 records. n: 300. Time: 54
Inserted 10000 records. n: 301. Time: 42
Inserted 10000 records. n: 302. Time: 42
Inserted 10000 records. n: 303. Time: 11
Inserted 10000 records. n: 304. Time: 9
Inserted 10000 records. n: 305. Time: 7
Inserted 10000 records. n: 306. Time: 8
Inserted 10000 records. n: 307. Time: 10
Inserted 10000 records. n: 308. Time: 10
Inserted 10000 records. n: 309. Time: 10
Inserted 10000 records. n: 310. Time: 10
Inserted 10000 records. n: 311. Time: 9
Inserted 10000 records. n: 312. Time: 11
Inserted 10000 records. n: 313. Time: 11
Inserted 10000 records. n: 314. Time: 13
Inserted 10000 records. n: 315. Time: 13
Inserted 10000 records. n: 316. Time: 12
Inserted 10000 records. n: 317. Time: 11
Inserted 10000 records. n: 318. Time: 10
Inserted 10000 records. n: 319. Time: 9
Inserted 10000 records. n: 320. Time: 12
Inserted 10000 records. n: 321. Time: 11
Inserted 10000 records. n: 322. Time: 12
Inserted 10000 records. n: 323. Time: 10
Inserted 10000 records. n: 324. Time: 11
Inserted 10000 records. n: 325. Time: 11
Inserted 10000 records. n: 326. Time: 13
Inserted 10000 records. n: 327. Time: 13
Inserted 10000 records. n: 328. Time: 12
Inserted 10000 records. n: 329. Time: 12
Inserted 10000 records. n: 330. Time: 10
Inserted 10000 records. n: 331. Time: 11
Inserted 10000 records. n: 332. Time: 10
Inserted 10000 records. n: 333. Time: 11
Inserted 10000 records. n: 334. Time: 11
Inserted 10000 records. n: 335. Time: 11
Inserted 10000 records. n: 336. Time: 10
Inserted 10000 records. n: 337. Time: 11
Inserted 10000 records. n: 338. Time: 12
Inserted 10000 records. n: 339. Time: 13
Inserted 10000 records. n: 340. Time: 14
Inserted 10000 records. n: 341. Time: 13
Inserted 10000 records. n: 342. Time: 10
Inserted 10000 records. n: 343. Time: 10
Inserted 10000 records. n: 344. Time: 9
Inserted 10000 records. n: 345. Time: 11
Inserted 10000 records. n: 346. Time: 12
Inserted 10000 records. n: 347. Time: 12
Inserted 10000 records. n: 348. Time: 10
Inserted 10000 records. n: 349. Time: 11
Inserted 10000 records. n: 350. Time: 13
Inserted 10000 records. n: 351. Time: 13
Inserted 10000 records. n: 352. Time: 11
Inserted 10000 records. n: 353. Time: 12
Inserted 10000 records. n: 354. Time: 10
Inserted 10000 records. n: 355. Time: 10
Inserted 10000 records. n: 356. Time: 12
Inserted 10000 records. n: 357. Time: 11
Inserted 10000 records. n: 358. Time: 11
Inserted 10000 records. n: 359. Time: 12
Inserted 10000 records. n: 360. Time: 11
Inserted 10000 records. n: 361. Time: 12
Inserted 10000 records. n: 362. Time: 13
Inserted 10000 records. n: 363. Time: 12
Inserted 10000 records. n: 364. Time: 11
Inserted 10000 records. n: 365. Time: 11
Inserted 10000 records. n: 366. Time: 11
Inserted 10000 records. n: 367. Time: 11
Inserted 10000 records. n: 368. Time: 12
Inserted 10000 records. n: 369. Time: 11
Inserted 10000 records. n: 370. Time: 14
Inserted 10000 records. n: 371. Time: 14
Inserted 10000 records. n: 372. Time: 15
Inserted 10000 records. n: 373. Time: 16
Inserted 10000 records. n: 374. Time: 18
Inserted 10000 records. n: 375. Time: 15
Inserted 10000 records. n: 376. Time: 11
Inserted 10000 records. n: 377. Time: 11
Inserted 10000 records. n: 378. Time: 11
Inserted 10000 records. n: 379. Time: 12
Inserted 10000 records. n: 380. Time: 11
Inserted 10000 records. n: 381. Time: 12
Inserted 10000 records. n: 382. Time: 12
Inserted 10000 records. n: 383. Time: 14
Inserted 10000 records. n: 384. Time: 12
Inserted 10000 records. n: 385. Time: 16
Inserted 10000 records. n: 386. Time: 14
Inserted 10000 records. n: 387. Time: 10
Inserted 10000 records. n: 388. Time: 12
Inserted 10000 records. n: 389. Time: 11
Inserted 10000 records. n: 390. Time: 11
Inserted 10000 records. n: 391. Time: 11
Inserted 10000 records. n: 392. Time: 12
Inserted 10000 records. n: 393. Time: 15
Inserted 10000 records. n: 394. Time: 16
Inserted 10000 records. n: 395. Time: 17
Inserted 10000 records. n: 396. Time: 13
Inserted 10000 records. n: 397. Time: 13
Inserted 10000 records. n: 398. Time: 17
Inserted 10000 records. n: 399. Time: 11
Inserted 10000 records. n: 400. Time: 16
Inserted 10000 records. n: 401. Time: 12
Inserted 10000 records. n: 402. Time: 13
Inserted 10000 records. n: 403. Time: 13
Inserted 10000 records. n: 404. Time: 13
Inserted 10000 records. n: 405. Time: 16
Inserted 10000 records. n: 406. Time: 10
Inserted 10000 records. n: 407. Time: 12
Inserted 10000 records. n: 408. Time: 14
Inserted 10000 records. n: 409. Time: 12
Inserted 10000 records. n: 410. Time: 18
Inserted 10000 records. n: 411. Time: 13
Inserted 10000 records. n: 412. Time: 20
Inserted 10000 records. n: 413. Time: 14
Inserted 10000 records. n: 414. Time: 15
Inserted 10000 records. n: 415. Time: 52
Inserted 10000 records. n: 416. Time: 22
Inserted 10000 records. n: 417. Time: 111
Inserted 10000 records. n: 418. Time: 81
Inserted 10000 records. n: 419. Time: 80
Inserted 10000 records. n: 420. Time: 78
Inserted 10000 records. n: 421. Time: 116
Inserted 10000 records. n: 422. Time: 123
Inserted 10000 records. n: 423. Time: 86
Inserted 10000 records. n: 424. Time: 24
Inserted 10000 records. n: 425. Time: 36
Inserted 10000 records. n: 426. Time: 21
Inserted 10000 records. n: 427. Time: 15
Inserted 10000 records. n: 428. Time: 17
Inserted 10000 records. n: 429. Time: 15
Inserted 10000 records. n: 430. Time: 15
Inserted 10000 records. n: 431. Time: 16
Inserted 10000 records. n: 432. Time: 15
Inserted 10000 records. n: 433. Time: 18
Inserted 10000 records. n: 434. Time: 20
Inserted 10000 records. n: 435. Time: 18
Inserted 10000 records. n: 436. Time: 20
Inserted 10000 records. n: 437. Time: 20
Inserted 10000 records. n: 438. Time: 22
Inserted 10000 records. n: 439. Time: 22
Inserted 10000 records. n: 440. Time: 26
Inserted 10000 records. n: 441. Time: 24
Inserted 10000 records. n: 442. Time: 69
Inserted 10000 records. n: 443. Time: 133
Inserted 10000 records. n: 444. Time: 17
Inserted 10000 records. n: 445. Time: 18
Inserted 10000 records. n: 446. Time: 20
Inserted 10000 records. n: 447. Time: 19
Inserted 10000 records. n: 448. Time: 19
Inserted 10000 records. n: 449. Time: 20
Inserted 10000 records. n: 450. Time: 18
Inserted 10000 records. n: 451. Time: 18
Inserted 10000 records. n: 452. Time: 19
Inserted 10000 records. n: 453. Time: 19
Inserted 10000 records. n: 454. Time: 19
Inserted 10000 records. n: 455. Time: 22
Inserted 10000 records. n: 456. Time: 33
Inserted 10000 records. n: 457. Time: 25
Inserted 10000 records. n: 458. Time: 23
Inserted 10000 records. n: 459. Time: 24
Inserted 10000 records. n: 460. Time: 134
Inserted 10000 records. n: 461. Time: 31
Inserted 10000 records. n: 462. Time: 21
Inserted 10000 records. n: 463. Time: 21
Inserted 10000 records. n: 464. Time: 21
Inserted 10000 records. n: 465. Time: 20
Inserted 10000 records. n: 466. Time: 25
Inserted 10000 records. n: 467. Time: 26
Inserted 10000 records. n: 468. Time: 24
Inserted 10000 records. n: 469. Time: 25
Inserted 10000 records. n: 470. Time: 31
Inserted 10000 records. n: 471. Time: 28
Inserted 10000 records. n: 472. Time: 31
Inserted 10000 records. n: 473. Time: 32
Inserted 10000 records. n: 474. Time: 33
Inserted 10000 records. n: 475. Time: 31
Inserted 10000 records. n: 476. Time: 40
Inserted 10000 records. n: 477. Time: 120
Inserted 10000 records. n: 478. Time: 23
Inserted 10000 records. n: 479. Time: 30
Inserted 10000 records. n: 480. Time: 33
Inserted 10000 records. n: 481. Time: 28
Inserted 10000 records. n: 482. Time: 26
Inserted 10000 records. n: 483. Time: 31
Inserted 10000 records. n: 484. Time: 22
Inserted 10000 records. n: 485. Time: 26
Inserted 10000 records. n: 486. Time: 25
Inserted 10000 records. n: 487. Time: 22
Inserted 10000 records. n: 488. Time: 38
Inserted 10000 records. n: 489. Time: 24
Inserted 10000 records. n: 490. Time: 27
Inserted 10000 records. n: 491. Time: 42
Inserted 10000 records. n: 492. Time: 39
Inserted 10000 records. n: 493. Time: 97
Inserted 10000 records. n: 494. Time: 59
Inserted 10000 records. n: 495. Time: 28
Inserted 10000 records. n: 496. Time: 24
Inserted 10000 records. n: 497. Time: 25
Inserted 10000 records. n: 498. Time: 24
Inserted 10000 records. n: 499. Time: 37
Inserted 10000 records. n: 500. Time: 24
Inserted 10000 records. n: 501. Time: 30
Inserted 10000 records. n: 502. Time: 31
Inserted 10000 records. n: 503. Time: 38
Inserted 10000 records. n: 504. Time: 65
Inserted 10000 records. n: 505. Time: 61
Inserted 10000 records. n: 506. Time: 47
Inserted 10000 records. n: 507. Time: 31
Inserted 10000 records. n: 508. Time: 38
Inserted 10000 records. n: 509. Time: 29
Inserted 10000 records. n: 510. Time: 31
Inserted 10000 records. n: 511. Time: 30
Inserted 10000 records. n: 512. Time: 33
Inserted 10000 records. n: 513. Time: 30
Inserted 10000 records. n: 514. Time: 35
Inserted 10000 records. n: 515. Time: 35
Inserted 10000 records. n: 516. Time: 33
Inserted 10000 records. n: 517. Time: 33
Inserted 10000 records. n: 518. Time: 30
Inserted 10000 records. n: 519. Time: 46
Inserted 10000 records. n: 520. Time: 75
Inserted 10000 records. n: 521. Time: 30
Inserted 10000 records. n: 522. Time: 36
Inserted 10000 records. n: 523. Time: 28
Inserted 10000 records. n: 524. Time: 29
Inserted 10000 records. n: 525. Time: 32
Inserted 10000 records. n: 526. Time: 34
Inserted 10000 records. n: 527. Time: 43
Inserted 10000 records. n: 528. Time: 38
Inserted 10000 records. n: 529. Time: 37
Inserted 10000 records. n: 530. Time: 29
Inserted 10000 records. n: 531. Time: 29
Inserted 10000 records. n: 532. Time: 30
Inserted 10000 records. n: 533. Time: 33
Inserted 10000 records. n: 534. Time: 39
Inserted 10000 records. n: 535. Time: 55
Inserted 10000 records. n: 536. Time: 73
Inserted 10000 records. n: 537. Time: 50
Inserted 10000 records. n: 538. Time: 54
Inserted 10000 records. n: 539. Time: 31
Inserted 10000 records. n: 540. Time: 38
Inserted 10000 records. n: 541. Time: 50
Inserted 10000 records. n: 542. Time: 40
Inserted 10000 records. n: 543. Time: 21
Inserted 10000 records. n: 544. Time: 38
Inserted 10000 records. n: 545. Time: 34
Inserted 10000 records. n: 546. Time: 41
Inserted 10000 records. n: 547. Time: 34
Inserted 10000 records. n: 548. Time: 31
Inserted 10000 records. n: 549. Time: 37
Inserted 10000 records. n: 550. Time: 58
Inserted 10000 records. n: 551. Time: 42
Inserted 10000 records. n: 552. Time: 36
Inserted 10000 records. n: 553. Time: 38
Inserted 10000 records. n: 554. Time: 55
Inserted 10000 records. n: 555. Time: 39
Inserted 10000 records. n: 556. Time: 45
Inserted 10000 records. n: 557. Time: 36
Inserted 10000 records. n: 558. Time: 48
Inserted 10000 records. n: 559. Time: 45
Inserted 10000 records. n: 560. Time: 36
Inserted 10000 records. n: 561. Time: 32
Inserted 10000 records. n: 562. Time: 36
Inserted 10000 records. n: 563. Time: 46
Inserted 10000 records. n: 564. Time: 51
Inserted 10000 records. n: 565. Time: 49
Inserted 10000 records. n: 566. Time: 47
Inserted 10000 records. n: 567. Time: 49
Inserted 10000 records. n: 568. Time: 45
Inserted 10000 records. n: 569. Time: 41
Inserted 10000 records. n: 570. Time: 39
Inserted 10000 records. n: 571. Time: 43
Inserted 10000 records. n: 572. Time: 68
Inserted 10000 records. n: 573. Time: 41
Inserted 10000 records. n: 574. Time: 33
Inserted 10000 records. n: 575. Time: 54
Inserted 10000 records. n: 576. Time: 39
Inserted 10000 records. n: 577. Time: 45
Inserted 10000 records. n: 578. Time: 50
Inserted 10000 records. n: 579. Time: 34
Inserted 10000 records. n: 580. Time: 36
Inserted 10000 records. n: 581. Time: 47
Inserted 10000 records. n: 582. Time: 59
Inserted 10000 records. n: 583. Time: 40
Inserted 10000 records. n: 584. Time: 37
Inserted 10000 records. n: 585. Time: 35
Inserted 10000 records. n: 586. Time: 59
Inserted 10000 records. n: 587. Time: 62
Inserted 10000 records. n: 588. Time: 26
Inserted 10000 records. n: 589. Time: 54
Inserted 10000 records. n: 590. Time: 54
Inserted 10000 records. n: 591. Time: 47
Inserted 10000 records. n: 592. Time: 53
Inserted 10000 records. n: 593. Time: 41
Inserted 10000 records. n: 594. Time: 45
Inserted 10000 records. n: 595. Time: 39
Inserted 10000 records. n: 596. Time: 45
Inserted 10000 records. n: 597. Time: 55
Inserted 10000 records. n: 598. Time: 50
Inserted 10000 records. n: 599. Time: 47
Inserted 10000 records. n: 600. Time: 39
Inserted 10000 records. n: 601. Time: 48
Inserted 10000 records. n: 602. Time: 43
Inserted 10000 records. n: 603. Time: 57
Inserted 10000 records. n: 604. Time: 38
Inserted 10000 records. n: 605. Time: 42
Inserted 10000 records. n: 606. Time: 44
Inserted 10000 records. n: 607. Time: 55
Inserted 10000 records. n: 608. Time: 46
Inserted 10000 records. n: 609. Time: 49
Inserted 10000 records. n: 610. Time: 44
Inserted 10000 records. n: 611. Time: 43
Inserted 10000 records. n: 612. Time: 43
Inserted 10000 records. n: 613. Time: 48
Inserted 10000 records. n: 614. Time: 36
Inserted 10000 records. n: 615. Time: 55
Inserted 10000 records. n: 616. Time: 58
Inserted 10000 records. n: 617. Time: 39

Is there any way to keep the performance linear at around 9-10 seconds per 10000 records?

score 0 · Answer 1 · answered Apr 04 '13 at 07:59

0

Your calculations concentrate on the key side (Will the keys fit into RAM?). That is important, but after reaching some database size (which is dominated by the value side in your case), the values written to disk will become more relevant for speed. I fear that you can only optimize your disk (any details?) in such a situation.

answered Apr 04 '13 at 07:59

Sven

496
4
9

Does KC keep the keys in the memory? Or does it fetch them from the disk for every write? I don't see any documentation mentioning that for HashDB, though they do mention it for B+ tree. – Tarandeep Gill Apr 04 '13 at 10:46
The system's total RAM usage is just around 1.8 GB, out of which the KC process is using about 1.2 GB. So I am assuming that the index is not being kept in the RAM. Is it true? How can I force it to keep the index in the RAM? – Tarandeep Gill Apr 04 '13 at 12:36
Please check the section "Effective Implementation of Hash Database" in the manual. It is a good idea to adjust mmap-size, at least in tokyocabinet. – Sven Apr 04 '13 at 14:27

MechanTOurS · Answer 2 · 2013-04-05T08:07:41.230

0

Did you modify msiz too?

Did you try a lower bnum parameter (something like 1 or 2 Billions)?

What is your filesystem (ext4 performed far better than ext3 during my tests)?

edited Apr 05 '13 at 08:07

answered Apr 04 '13 at 12:53

MechanTOurS

1,485
2
15
18

msiz, as I mentioned is 2 GB. I changed bnum to 1 Billion, and its still the same. – Tarandeep Gill Apr 04 '13 at 16:01
Can you clean up some confusion by editing? In you question you said msiz is 30 GB.If so: To rule out overflow errors, could you try something safe like 1 or 2 GB? – Sven Apr 05 '13 at 06:32

Kyoto Cabinet HashDB inserts are not O(1)

2 Answers2