Read access violation for memory mapped vector in debug mode

Question

While attempting to use boost::interprocess for storing a std::vector in a memory mapped file, I am getting the exception Exception thrown: read access violation. when I try to push back on a loaded vector, but only in debug mode.

This minimal example code (written by @sehe) is retrieved from https://stackoverflow.com/a/29602884/2741329, and it crashes on MSVC14 in debug mode and executed more than once:

#include <boost/interprocess/managed_mapped_file.hpp>

namespace bi = boost::interprocess;

int main() {
    std::string vecFile = "vector.dat";
    bi::managed_mapped_file file_vec(bi::open_or_create,vecFile.c_str(), 1000);

    typedef bi::allocator<int, bi::managed_mapped_file::segment_manager> int_alloc;
    typedef std::vector<int, int_alloc>  MyVec;

    MyVec * vecptr = file_vec.find_or_construct<MyVec>("myvector")(file_vec.get_segment_manager());

    vecptr->push_back(rand());
}

EDIT:

This is the Visual Studio error message:

Here the point where the exception happens:

This is the call stack (click on the pic to enlarge it):

It would be helpful to show the stack trace when it crashes. — John Zwinck, Jul 14 '18 at 02:43
Putting C++ std template types into shared memory is a very risky endeavor. You have a raging dependency on the compiler/library version and the build settings. An easy way to trigger this crash is having code built with debug settings on one side and code built with release settings on the other. It bombs in the library code that implements the iterator debugging feature, it changes the object layout. Sayonara when the data was generated by code that was build with iterator debugging turned off. Keep it simple. — Hans Passant, Jul 14 '18 at 12:29
@HansPassant, I am not using it for interprocess communication. My use case is to just store a huge STL vector on a memory mapped file. (And I am fine with having to debug and release builds not playing well together.) BTW, how do you suggest to deal with my use case? — gmas80, Jul 14 '18 at 13:40
The `open_or_create` is pretty confuzzling. That better always be "create" or you are going to have a bad day. If IPC is not the intention then the true intention gets to be pretty mysterious. A plain std::vector already uses a memory mapped file without any help. The paging file. Don't do this. — Hans Passant, Jul 14 '18 at 13:46
@HansPassant, my intention would be to take advantage of the memory file-mapping mechanism for file persistence. In other words, I don't have to provide a separated mechanism to serialize/deserialize the vector for future use. Does it make sense? — gmas80, Jul 14 '18 at 13:58
Storing a raw std::vector in a file is not a good idea. You just can't ever read it back reliably, it contains pointers that are only valid once and become junk when the process terminates. It doesn't even contain the element data, stored on the heap and not the mmf. You can save a plain T[] and have some hope, as long as T itself doesn't change and doesn't store pointers. — Hans Passant, Jul 14 '18 at 14:08
Doesn't `bi::allocator` manage the element data allocation on the mmf? — gmas80, Jul 14 '18 at 14:12
@HansPassant with all due respect, this is spreading FUD. I understand you're simply unaware of the Boost Interprocess library, but the usage here is perfectly fine (you have a valid intuition with respect to racing around `open_or_create` but that's clearly not the issue here and unrelated to the question). For the OP: I'll look at this layer when I'm around a computer. (Are you by any chance running multiple copies simultaneously?) — sehe, Jul 14 '18 at 15:10
@sehe, thank you for the clarification. I have also opened a more general ticket about using STL containers with `boost::interprocess` here: https://github.com/boostorg/interprocess/issues/58 . (I am not running multiple copies simultaneously) — gmas80, Jul 14 '18 at 15:43

sehe · Accepted Answer · 2018-07-15T16:02:16.747

As a brainwave, disable MSVC debug iterators.

I'm not sure how (because iterators aren't persisted?) but somehow iterator debugging might add raw pointers inside the memory layout of the std::vector - violating standard library assumptions about allocator use.

Test Results

Creating a VM on azure just for the purpose, tested with the following slightly modified code to better understand the crash reasons:

#include <boost/interprocess/managed_mapped_file.hpp>
#include <iostream>

namespace bi = boost::interprocess;

int main() {
    std::string vecFile = "vector.dat";
    //std::remove(vecFile.c_str());
    std::cout << __LINE__ << "\n";
    {
        bi::managed_mapped_file file_vec(bi::open_or_create, vecFile.c_str(), 100000);

        typedef bi::allocator<int, bi::managed_mapped_file::segment_manager> int_alloc;
        typedef std::vector<int, int_alloc>  MyVec;

        MyVec * vecptr = file_vec.find_or_construct<MyVec>("myvector")(file_vec.get_segment_manager());

        vecptr->push_back(rand());
        std::cout << "size: " << vecptr->size() << "\n";
    }
    std::cout << __LINE__ << "\n";
    {
        bi::managed_mapped_file file_vec(bi::open_or_create, vecFile.c_str(), 100000);

        typedef bi::allocator<int, bi::managed_mapped_file::segment_manager> int_alloc;
        typedef std::vector<int, int_alloc>  MyVec;

        MyVec * vecptr = file_vec.find_or_construct<MyVec>("myvector")(file_vec.get_segment_manager());

        vecptr->push_back(rand());
        std::cout << "size: " << vecptr->size() << "\n";
    }
    std::cout << __LINE__ << "\n";
}

Reproduces the issue. First run:

Subsequent run (with the std::remove line commented as shown):

WORKAROUND DEMO

After putting

#define _ITERATOR_DEBUG_LEVEL 0

at the very top AND REMOVING THE vector.dat file because the change alters binary layout:

Note: in your actual project you may require putting that #define in multiple translation units (especially consider stdafx.cpp). It's probably much better to include it in the project property sheets so it applies to all (future) translation units!

I have added `#define _ITERATOR_DEBUG_LEVEL 0`, but I am still getting the same exception and call stack. — gmas80, Jul 14 '18 at 15:45
I completely forgot how annoying everything is on Windows... https://i.imgur.com/HEF95FD.png This is taking a lot of time — sehe, Jul 15 '18 at 14:47
I've just checked it myself. It is indeed the iterator debugging and indeed putting the define (in all the appropriate places!) removes the issue. Don't forget to actually remove the `vector.dat` file in between runs when you change the buld flags/configuration! — sehe, Jul 15 '18 at 15:55
Amended the answer with actual proof and also various hints at what could go wrong and why you need to delete the `vector.dat`. To the downvoter, appreciate if you reconsider. — sehe, Jul 15 '18 at 16:03

Read access violation for memory mapped vector in debug mode

1 Answers1

Test Results

WORKAROUND DEMO

Linked