1

Is there a way to reduce the thread contention with the xodus garbage collector?

I've been trying to set up an implementation where I use multiple environments to reduce contention on writes, which works to some extent - however, when the garbage collector runs, it still seems to block any write thread across ALL environments... I can observe that all of my write threads get blocked every single time the garbage collector runs.

Doesn't the garbage collector in xodus collect garbage per environment?

Any other tuning tips for xodus would be appreciated... I'm having a lot of trouble getting much write performance out of it. The only environment config variable that seems to help at all, is to bump up the log file size slightly. Most other things, just don't seem to have any impact.

I've tried running with the garbage collector off, but then the file counts go crazy quite quickly, which causes other issues.

user2163960
  • 1,871
  • 19
  • 22
  • I've also tried disabling the shared log cache, which gets me a thread per enviornment... but that doesn't seem to help the performance either, I'm guessing due to other penalties. Also, since it doesn't use a thread pool for the garbage collectors, the number of threads explodes. – user2163960 Jun 26 '18 at 22:13
  • What reading threads are blocked on? Can you take a jstack thread dump? – Vyacheslav Lukianov Jul 06 '18 at 08:50
  • I'm primarily in a bulk / batch write mode for the timings I'm doing. There may be a few reads here and there, but not enough to be noticeable in the timings. For comparison, I wrote a layer that put the same software on top of the MVStore from the H2 project, and my process time to load a dataset is about 8 minutes. With Xodus, it is 30+. Primarily, because MVStore allows you to write without a commit (it autocommits in the background) which is fine for my use case of bulk loading data. – user2163960 Jul 07 '18 at 19:16
  • How often do you commit/flush your writing transactions? The more seldom flushes the better overall writing performance. – Vyacheslav Lukianov Jul 10 '18 at 14:42
  • With the way the transaction wrappers go, I end up with a transaction per write... but I'm writing millions of objects. In general, I have a pool of threads reading data, transforming it, and trying to write it out one object at a time. To reduce the contention among the threads, I hashed the environments out, so that each of my threads is likely to be hitting a different env - but during bulk load, they are all just loading data as fast as they can. I'm guessing each transaction is flushing... so they are slow. – user2163960 Jul 12 '18 at 03:05

1 Answers1

1

Xodus' GC runs per environment in a single thread with the name "Exodus shared background cleaner". Each GC cycle can only be performed against a single environment, so garbage collection of different environments is performed successively. That's why there should not be any contention between environments. In addition, read-only transactions don't block on any writing transactions, GC transactions as well.

Some configuration parameters (see EnvironmentConfig class) that can affect GC timings:

  • Try a value of exodus.gc.minUtilization less that 50 which is default value. This configuration parameter sets the target percent of "useful space" in the database. The lower value the less GC load.
  • Try a value of exodus.gc.transactionAcquireTimeout less than 1000 which is default value. This configuration parameter sets the number of milliseconds during which GC tries to acquire exclusive transaction to reclaim data. A lower value decreases the probability of GC being active during peak write load.
  • Try a value of exodus.gc.transactionTimeout less than 1000 which is default value. This configuration parameter sets the number of milliseconds during which GC reclaims data in an exclusive transaction if GC was lucky to acquire it.
  • Finally, you can try to set exodus.gc.useExclusiveTransaction=false, then GC will always try to reclaim data in an optimistic loop, but our tests show that optimistic loop almost always works worse under high load.
Vyacheslav Lukianov
  • 1,913
  • 8
  • 12