How do I fix this IO error with rocksdb in Python?

Question

I'm making a Flask app with rocksdb as the key-value store. I'm able to create and add to the database when I run the app once, but on restarting the app, I get this error:

rocksdb.errors.Corruption: b"Corruption: Can't access /000010.sst: IO error: No such file or directorywhile stat a file for size: database_name.db/000010.sst: No such file or directory\n"

I tried the answer from this link: rocksdb.errors.RocksIOError: IO error: While lock file: sample.db/LOCK: Resource temporarily unavailable but it didn't help.

This is the code:

opts = rocksdb.Options()
opts.create_if_missing = True
opts.wal_dir = absolute path to database
opts.db_log_dir = absolute path to database
opts.wal_ttl_seconds = 100
os.system('rm database_name.db/LOCK')
database_name_db = rocksdb.DB('database_name.db', opts)

what does your directory look like? What files are in it? I think you messed up your DB — Asad Awadia, Jul 28 '22 at 03:30
There's a log file, sst file, text files called: CURRENT, IDENTITY, LOCK, LOG, MANIFEST, and OPTIONS. There isn't much documentation, but is there a way to get a database from the path? Otherwise I have to delete the database before re-running the app. I tried running the app as a daemon by using nohup, but it times out sporadically. — Swashbuckler, Jul 29 '22 at 20:31
First thing is to get rid of `os.system('rm database_name.db/LOCK')` you are not to remove the lock file yourself. I suspect you added this because you didn't close the db properly. Make sure the DB is closed upon exit. Wdym database from a path? You don't have to delete the DB to get it to work. What are you trying to achieve just a basic app with rocksdb? — Asad Awadia, Jul 30 '22 at 13:17
When I deleted that line, I got this error: ``` rocksdb.errors.RocksIOError: b'IO error: While lock file: database_name.db/LOCK: Resource temporarily unavailable' ``` The way my app works is my post request does calculations and keeps the results in a key value store (rocksdb). The get request uses the key as part of the url to display the value to the web page. I'm developing in a VM and running the app as a nohup process, but that still times out. — Swashbuckler, Jul 31 '22 at 22:32
Yeah because you aren't closing the DB properly when your app exits. That's why the lock file sticks around. That is a simple use case and should work very easily with rocksdb - can you push your full code to github? — Asad Awadia, Aug 01 '22 at 02:59
I can't push it to github because it's an internal company project. I call database_name_db.close() at the end when the app exits. I'm using python-rocksdb, btw. — Swashbuckler, Aug 01 '22 at 19:53
can you try `rocksdb.DB('accuracy.db', rocksdb.Options(create_if_missing=True))` and see if that works first? basically get rid of all the options - specially the wal directory option — Asad Awadia, Aug 02 '22 at 16:39

How do I fix this IO error with rocksdb in Python?

0 Answers0