0

STXXL 1.4.1 on Windows 10 / VS2015.

I have some code in which I essentially use this:

typedef stxxl::vector<RC, 4, stxxl::lru_pager<8>, 2 * 1024> TVector;
typedef stxxl::deque<RC, TVector> CellIndexList;

This is to lower the block size from the default of 2*1024*1024; I had to do this because I have

std::vector<CellIndexList> list(400);

And since the default stxxl::vector allocates a buffer of ~60MB, I had 60*400 MB in my vector, which crashed with bad_alloc (32 bits) - so that's how I came to my current setup.

With this as background, what I'm now running into is that my code crashes with the following stxxl diagnostics:

[STXXL-ERRMSG] Exception thrown in ~vector()

And when I add some additional diagnostics to stxxl::vector::~vector(), I get the following:

Dynamic exception type: class std::system_error
std::exception::what: resource unavailable try again: resource unavailable try again

So, the flush() in the stxxl::vector destructor fails with this exception; from googling it seems that this exception means that the maximum number of threads has been reached.

Now I'm struggling to understand why there would be extra threads created when doing the flush(). I have looked around the stxxl source code and I've found some mentions of different types of disk queues, and I have also tried to enable STXXL_HACK_SINGLE_IO_THREAD in defines.h but all of that is just trying random things as I'm lacking an overview of how many threads stxxl creates for which purposes and under what circumstances.

I've also tried making a standalone problem that exhibits this problem but failed; I can easily do

std::vector<CellIndexList> d(10000);

and fill them with lots of data without issue. So there must be something specific about the way I'm using things in my real code that is causing this threading problem. It's just too large to start chopping things out until I can make the problem go away.

So, my question: is there any place in stxxl where potentially large numbers of threads are being created? Is there anything else I can try to find the problem here?

Roel
  • 19,338
  • 6
  • 61
  • 90
  • `std::vector` is not supposed to have any `flush` or deal with threads, this question is extremely vague... – user7860670 Oct 23 '18 at 11:32
  • Oh right I see how that can be confusing because of the typedef - it's a std::vector>, and stxxl::deque<> is implemented in terms of stxxl::vector<>, and it's the destructor of that vector that I'm talking about. So this has little to do with std::vector<>, but with stxxl::vector<>. – Roel Oct 23 '18 at 12:23
  • Did you copy/paste that error message correctly? Sounds a lot more like "Resources temporarily unavailable". Not so sure it is a good idea to focus on threads with this particularly library, I'd look for a running out of swap space first. – Hans Passant Oct 23 '18 at 13:50
  • Yes it's copy/paste; others report it e.g. here: https://stackoverflow.com/questions/42315723/c-error-when-creating-threads-in-loops . I've already increased my page file size to 75gb which should be enough for my particular data set with the error. Thanks for thinking along. – Roel Oct 23 '18 at 14:25

0 Answers0