1

I'm about to begin work on a new project on a Raspberry Pi 3. It will be controlled with a complex GUI so interactive performance is important.

I decided to use LMDB for persistence, as - outside performance - its special traits make my system a lot simpler and for me there's no downside. The application will be written in Rust, using the lmdb crate.

The critical path will be a single-threaded part where I will get the current timestamp and write a total of 16 or 20 bytes (not sure yet, is 16 bytes really better?) to the database under a key I already have computed. For doing so (beginning with timestamp and ending after the write transaction has been commited) I have a performance budget of 2 milliseconds.

As far as I heard writes are LMDB's worst criteria, and those are very small random writes, so this is probably the worst possible application. This thought made me ask this question.

Further information: as this is a GUI application this path will be called at most 100 times per second, and also never more than 1000 times per hour (this is a rewrite from scratch, and those are 10 times the numbers measured in the current system).

I do not understand much about how LMDB works but AFAIU it is using memory mapped files. I was hoping this means the OS internals will write back the pages aggregated.

Is 2ms a realistic goal for such an application? What would I need to consider to keep myself inside this window? Do I need to cache those writes manually?

5-to-9
  • 649
  • 8
  • 16
  • 1
    Why did you tag Rust? – BurntSushi5 Aug 07 '17 at 17:50
  • It's implemented in Rust. I was not sure if that was a useful information but I wanted to at least somehow include the information. I edited it now to make it clear. – 5-to-9 Aug 07 '17 at 19:08
  • 1
    I think it would help if you include some sample code. I feel like the obvious answer to this question is to benchmark it. Have you tried that? I don't see how anyone could answer this question for you because it at least depends in part on the hardware you're running it on, no? – BurntSushi5 Aug 07 '17 at 23:21
  • how did you compute performance budget? Is this 2 ms on average or every time? What if db/os only flushes to disk once in a while? disk writes can be expensive, so my guess is no, not realistic. – ren Aug 07 '17 at 23:46

1 Answers1

0

It depends on how much data you have in you LMDB.

2ms is definitely achievable for small DB. I have got over 100K writes per second with SSDs and small DB size (<1GB). But if the DB is larger than your memory, you probably should not use LMDB if you are worried about write performance.

To wrap up, if your DB is smaller than memory, you can go ahead and use LMDB, 2ms is definitely enough for a write operation. If your DB is larger than memory, then you better use LSM-based kv-stores like LevelDB or RocksDB.

hjk41
  • 239
  • 2
  • 10