1

I'm writing an application that requires me to store a very large map with key-value pairs. So relying solely on a Golang map in memory will not cut it. BoltDB looks like it might be the right thing for this scenario.

My application will continuously write and delete (+retrieve the deleted value) values on a single key-value store. For performance reasons I want to avoid a new BoltDB db.Update every time I write a value as this would also write to disk. However, I want to periodically flush the in-memory state back to the file the BoltDB relies on to avoid running out-of-memory.

How can this be achieved with BoltDB? Is it feasible to create a single BoltDB transaction (db.Update) for my application and call tx.Commit periodically to flush to disk?

Roper
  • 903
  • 1
  • 9
  • 17
  • Sure, if you're OK with the potential data loss. Anything that has not been flushed will be lost if the program exits due to a crash, system failure, etc. – Adrian Aug 28 '17 at 13:59
  • Did you experience actual performance problems? Bolt is quite fast. And in case you have, are you positively sure that they are stemming from Bolt? – Markus W Mahlberg Aug 28 '17 at 17:24
  • Thanks for your responses. I was asking out of precaution because I have an insanely work-heavy task at hand that is likely to run for days. Therefore I want to get as much performance as possible and committing each write seemed to be a very bad idea to me. Now I've started experimenting with Bolt and it seems to run fine. The only issue I'm facing is that with a growing DB size that now exceeds my memory, the system is constantly swapping. However, this may very well be totally unrelated to Bolt and a memory leak of my application. – Roper Aug 29 '17 at 17:15
  • Bolt uses memory mapped files so that makes sense – Xeoncross Nov 08 '19 at 03:53

0 Answers0