0

I want to write a lot of data to a lmdb data base with several named (sub) data bases. I run into the following problem:

  • To write to one named data base, I need to open a transaction for this named data base.
  • This implies: To write to another named data base, I need to open a different transaction.
  • Two write transaction inside the same main data base cannot exist at the same time.
  • This implies: I need to commit and close a transaction each time I want to switch from writing to one named data base to writing to another named data base.
  • Creating and committing write transactions is a really slow operation.

I rather would like to keep one long-running write transaction for all write operations and commit it once --- when all the work is done.

Is this possible with lmdb (if yes, at which point did I err in my analysis)?

Yaakov Belch
  • 4,692
  • 33
  • 39

2 Answers2

1

You can open as many named databases within the same write transaction as you like.

So:

  • Open write transaction
  • Open named databases as required and write to them
  • Commit your transaction

As long as you take into account that you can only ever have one write-transaction at a time (read-only transactions are no problem), and that your other transactions will only see the result of your write-transaction once you commit, you can of course have one long-running write transaction.

cmollekopf
  • 131
  • 3
  • The problem with this approach is: Each time I want to write to a different sub-database, I need to commit the current transaction. This involves lock/unklock operations on disk files and is inherently slow. My question was whether I can run **all** write operations without the overhead of opening and closing many transactions – Yaakov Belch Feb 27 '17 at 14:35
  • You don't need to commit between writing to different databases. The transaction is valid for the complete database (so all your named sub-databases). You can open as many as you want within the same transaction, and once you're done commit once. – cmollekopf Mar 01 '17 at 14:56
  • I thought so as well. But it didn't work: Whenever you want to open a sub-data-base, you need to create a transaction for this sub-data base. That's because of the API: you can't access a sub-data-base except through its own transaction object. More than one write transactions clash and the process hangs. This was the reason for me to ask this question... – Yaakov Belch Mar 01 '17 at 16:44
  • I think now that lmdb.open(..., lock=False) solves this issue. – Yaakov Belch Mar 01 '17 at 16:47
1

you err in your analysis at this point

  • This implies: To write to another named data base, I need to open a different transaction.

One transaction handle can be used to open multiple subdatabases in an lmdb environment.

Note: a single transaction can open multiple databases. Ref

sidnt
  • 170
  • 10