2

A RocksDB newbie here. I am trying to load data into newly created RocksDB databases. I am using the RocksDB Sharp .NET wrapper around the native Windows RocksDB lib.

Everything seems to be working, but when I peek inside the directories it creates, I only see log files (always 000003.log). I see my data inside, but I thought it is supposed to create something other than the logs. When I try opening it with FastNoSQL, it worked, but, for some reason, I see SST files being created. (Which is probably the normal format.)

I am using WriteBatchWithIndex and I tried setting plain table / block table options. Nothing changes.

I tried tweaking PrepareForBulkLoad, SetAllowMmapWrites and whatnot. The compression is off.

Is that normal?

EDIT: I see that the bigger tables (over 10 Mb) do get SST files. Is there a way to force the smaller ones to "commit" the logs?

Vadim Berman
  • 1,932
  • 1
  • 20
  • 39

2 Answers2

2

OK, figured it myself.

The easiest way to force RocksDB to create SST immediately was to disable WAL (Write-Ahead Log).

To the helpful souls that didn't bother answering the question but will want to comment that the approach is wrong: yes, I'm aware of the dangers. In my situation, it's OK. It's a special generation-only project working on the files exclusively, and the results will be distributed to another application.

Vadim Berman
  • 1,932
  • 1
  • 20
  • 39
2

There is an explicit Flush call on rocksdb::DB

  // Flush all mem-table data.
  virtual Status Flush(const FlushOptions& options,
                       ColumnFamilyHandle* column_family) = 0;
  virtual Status Flush(const FlushOptions& options) {
    return Flush(options, DefaultColumnFamily());
  }

Turning off the WAL also means sacrificing crash-safety (for records only written to the memtables), which may be OK for your use-case, but in general I would not recommend it. RocksDb will retire and clean up the persisted WAL relatively aggressively anyway, unless you configure it not to.

midor
  • 5,487
  • 2
  • 23
  • 52
  • Thanks, @midor. I actually tried Flush() before, but it didn't cause the SST files to be created for some reason. Maybe I should have defined the options. – Vadim Berman Mar 23 '18 at 06:00
  • 1
    `Flush` has a single options `wait`, which defaults to true. I have used this many times and it has always immediately written memtable contents to disk for me, but rocksdb has so many options, I can't point to anything you could have done wrong. – midor Mar 23 '18 at 07:02
  • What did you mean by "RocksDb will retire and clean up the persisted WAL relatively aggressively anyway, unless you configure it not to."? It seems that this thing is biting us in the posterior when we tried upgrading: https://stackoverflow.com/questions/74507180/rocksdb-7-9-0-library-cant-open-rocksdb-6-2-2-files – Vadim Berman Nov 20 '22 at 10:40