bsddb3 DB.get/put hangs with transaction

Question

I'm fairly new to Berkeley DB, and I'm attempting to use it in Python with bsddb3 with transactions for power-safety With DB_AUTO_COMMIT and no transaction argument reads and writes work just fine. But the moment I call get/put with a manual transaction the call hangs indefinitely, using virtually no CPU (about 50k cycles/sec) and performing no disk I/O.

    _data_store_env.log_set_config(bdb.DB_LOG_AUTO_REMOVE, True)
    _data_store_env.set_lg_max(256 * 2**20)
    _data_store_env.set_cachesize(0, 512 * 2**20)
    _data_store_env.set_lg_dir(str(_journal_path))
    _data_store_env.set_tmp_dir(str(tmp_dir))
    _data_store_env.open(str(_bulk_data_path), bdb.DB_CREATE | bdb.DB_INIT_LOCK | bdb.DB_INIT_LOG | bdb.DB_INIT_MPOOL | bdb.DB_INIT_TXN | bdb.DB_RECOVER | bdb.DB_THREAD)

    # Originally I simply used DB_AUTO_COMMIT, but I changed it to see if this way would fix the hang. It didn't.
    txn = _data_store_env.txn_begin()

    _data_store = bdb.DB(_data_store_env)
    _data_store.set_flags(bdb.DB_CHKSUM)
    _data_store.set_pagesize(65536)
    _data_store.open("filestore.db", None, bdb.DB_HASH, bdb.DB_CREATE | bdb.DB_THREAD, 0x660, txn)

    _idx_store = bdb.DB(_data_store_env)
    _idx_store.set_flags(bdb.DB_CHKSUM | bdb.DB_DUPSORT)
    _idx_store.open("idxstore.db", None, bdb.DB_HASH, bdb.DB_CREATE | bdb.DB_THREAD, 0x660, txn)
    _data_store.associate(_idx_store, lambda key, data: key[0:9], bdb.DB_IMMUTABLE_KEY, txn)

    txn.commit()
...
    # It doesn't matter whether this flag is present or not. Both produce the same result.
    txn = _data_store_env.txn_begin(None, bdb.DB_TXN_BULK)
...
    # Never returns
    file_exists = _idx_store.has_key(entry_key, txn)
...
    # Also never returns
    _data_store.put(file_hash, file_data, txn)

Am I doing something wrong? Do transactions even work in bsddb3?

Justin Olbrantz · Answer 1 · 2018-04-21T04:38:34.403

After a number of hours debugging with a native code debugger, I believe I've gotten fairly close to a root cause. It appears that the hang does not occur in the get/put call, but after it - in a call to len on the database object performed by the debugger GUI after the get/put call completes (as well as periodically if the program is running free). This is apparently getting stuck in a resource acquisition loop that never completes because a transaction is in progress (I don't have symbols for the build of bsddb3 - which statically links to BDB, no less - so I'm not interested in spending the time to figure out what the dozen or so functions in the call stack beneath the DB_length call are). This is likely also related to the fact that the response time to any debugger commands increased to like 5 seconds (when it's not deadlocked) when I introduced bsddb3 to the project.

In summary, this appears to be a toxic interaction between the debugger GUI trying to keep its variables window up to date and the design decision that bsddb3 DBs should be mappings. I am very much open to suggestions how to work around this; I've tried a few things but have been unable to figure out how to delete the len method from the DB objects (it's a read-only property since it's implemented in C). Right now I'm creating a wrapper around the DB which is NOT a sequence; that fixes the problem unless you hover over the DB object accidentally and the debugger tries to fetch its properties.

bsddb3 DB.get/put hangs with transaction

1 Answers1