Java Berkeley DB is used in my system to store persistent data.
Since I have a large amount of data to be loaded, I attempt to do that with a number of threads. When the number of threads is low, e.g., 10, it works fine. However, when it is set to a higher value, e.g., 30, the reading processes get stuck. It looks like the Java Berkeley DB has an upper limit for concurrency reading? Am I right? How would I update the limit?
Did you say ... "to be loaded?"
Uh huh, thought you did!
Therefore, the threads that you speak of are not "reading" threads: they are "writing" threads!
And, guess what: they are competing with one another, and, guess what, they are losing!
Unfortunately, your "attempt" to speed things up through the use of threads was (IMHO ...) "sincere, but misguided." Ultimately, a Berkely DB is "a single, on-disk data structure," and so there are (IMHO...) no opportunities for speeding-up the process through the use of multi-threading.
Various other strategies, though, might work. For example, you might find that if you sort the records that are to be inserted, through some appropriate external command, the process of inserting those records might well become “usefully (much?) faster.” Enough of a speed-difference, in other words, to more-than(!) make up for the time spent sorting. (However, there will only be one way to find out if this is true in your situation: "benchmarking, your using actual data, your actual sort-command, and so on.")
