Memory mapped file std::allocator implementation freezes WM6 device

Question

I have a Visual Studio 2008 C++ project for Windows Mobile 6.x where I need more memory than is available to me in the 32MB process slot. So, I'm looking at using memory mapped files. I've created a standard allocator implementation that replaces new/delete with CreateFileMapping and MapViewOfFile.

The intended use is something like this:

struct Foo
{
    char a[ 1024 ];
};

int _tmain( int argc, _TCHAR* argv[] )
{
    std::vector< boost::shared_ptr< Foo > > v;
    for( int i = 0; i < 40000; ++i )
    {
        v.push_back( boost::allocate_shared< Foo >( MappedFileAllocator< Foo >() ) );
    }
    return 0;
}

With the std::allocator, I can get 28197 iterations in that example before I get a std::bad_alloc exception. With the MappedFileAllocator, I get 32371 iterations before the device completely freezes and has to be rebooted. Since my device has 512MB of RAM, I expected to be able to get far more iterations out of that loop.

My MappedFileAllocator implementation is:

template< class T >
class MappedFileAllocator
{
public:
    typedef T         value_type;
    typedef size_t    size_type;
    typedef ptrdiff_t difference_type;
    typedef T*        pointer;
    typedef const T*  const_pointer;
    typedef T&        reference;
    typedef const T&  const_reference;

    pointer address( reference r ) const { return &r; };
    const_pointer address( const_reference r ) const { return &r; };

    /// convert a MappedFileAllocator<T> to a MappedFileAllocator<U>
    template< class U >
    struct rebind { typedef MappedFileAllocator< U > other; };

    MappedFileAllocator() throw() : mapped_file_( INVALID_HANDLE_VALUE ) { };

    template< class U >
    explicit MappedFileAllocator( const MappedFileAllocator< U >& other ) throw()
        : mapped_file_( INVALID_HANDLE_VALUE )
    {
        if( other.mapped_file_ != this->mapped_file_ )
        {
            ::DuplicateHandle( GetCurrentProcess(), 
                other.mapped_file_,
                GetCurrentProcess(),
                &this->mapped_file_,
                0,
                FALSE,
                DUPLICATE_SAME_ACCESS );
        }
    };

    pointer allocate( size_type n, const void* /*hint*/ = 0 )
    {
        if( n > max_size() )
           throw std::bad_alloc();

        if( n > 0 )
        {
            size_type buf_size = n * sizeof( value_type );
            mapped_file_ = ::CreateFileMapping( INVALID_HANDLE_VALUE, 
                NULL,
                PAGE_READWRITE,
                0,
                buf_size,
                L"{45E4FA7B-7B1E-4939-8CBB-811276B5D4DE}" );

            if( NULL == mapped_file_ )
                throw std::bad_alloc();

            LPVOID f = ::MapViewOfFile( mapped_file_, 
                FILE_MAP_READ | FILE_MAP_WRITE, 
                0, 
                0, 
                buf_size );

            if( NULL == f )
            {
                ::CloseHandle( mapped_file_ );
                mapped_file_ = INVALID_HANDLE_VALUE;
                throw std::bad_alloc();
            }
            return reinterpret_cast< T* >( f );
        }

        return 0;
    };

    void deallocate( pointer p, size_type n )
    {
        if( NULL != p )
        {
            ::FlushViewOfFile( p, n * sizeof( T ) );
            ::UnmapViewOfFile( p );
        }
        if( INVALID_HANDLE_VALUE != mapped_file_ )
        {
            ::CloseHandle( mapped_file_ );
            mapped_file_ = INVALID_HANDLE_VALUE;
        }
    };

    size_type max_size() const throw() 
    { 
        return std::numeric_limits< size_type >::max() / sizeof( T );
    };

    /// handle to the memory-mapped file
    HANDLE mapped_file_;

private:

    /// disallow assignment
    void operator=( const MappedFileAllocator& );

}; // class MappedFileAllocator

Can anybody suggest where I may be going wrong with my MappedFileAllocator implementation?

Thanks, PaulH

check if each pointer returned from allocate() is aligned on some boundary; it looks as if MapViewOfFile may consume a single page each time you try to map the file. — vividos, Jun 28 '11 at 11:53
@vividos - They're ARM aligned on 4-byte boundaries. The WM version of MVOF doesn't require page alignment. http://msdn.microsoft.com/en-us/library/aa914405.aspx — PaulH, Jun 29 '11 at 18:33
Then I don't know what's the problem. Next thing I would try would be a memory pool in LMA allocated by VirtualAlloc(), instead of using anonymous file mapping. This document may help: http://davidfindlay.org/weblog/files/ce_lma.php — vividos, Jun 30 '11 at 06:59
@vividos - That's beautiful. I'm going to give that a shot. Thanks! — PaulH, Jun 30 '11 at 14:30

score 1 · Answer 1 · answered Jul 26 '11 at 18:10

Check if boost::interprocess supports Windows Mobile. They have facilities for creating memory-mapped files and using the memory to allocate any piece of data you want:

http://www.boost.org/doc/libs/1_47_0/doc/html/interprocess/sharedmemorybetweenprocesses.html#interprocess.sharedmemorybetweenprocesses.mapped_file

score 0 · Accepted Answer · answered Apr 09 '14 at 13:45

I just got the "popular question" badge for this one, so I thought I'd (belatedly) post the answer. I was running out of available handles. There is an unsigned short counter in the Windows Mobile kernel relating to handle allocation that was overflowing. Once that happened, the device would freeze.

score 0 · Answer 3 · answered May 04 '11 at 23:30

0

You're using an anonymous file mapping (one without an actual file backing it). When you do this, the mapping is backed by the system page file. It is likely that the Mobile OS does not actually have a Page File because the "hard disk" is probably a flash device. It's generally a bad idea to do virtual memory paging to flash devices because the nature of virtual memory means a high volume of writes which can quickly burn out flash (especially older types).

This seems to be backed up by the number of iterations you're getting. It looks like you're able to map up to around 60% of the total memory on your device.

You can probably get this to work with larger sections if you open a real file (using OpenFile) and map that rather than using INVALID_FILE_HANDLE in CreateFileMapping. But be careful of your flash storage (and performance!) if you're going to be writing to your file mapping very much.

answered May 04 '11 at 23:30

SoapBox

20,457
3
51
87

The Windows Mobile Large Memory Area is nearly 1GB. At best, I was only able to consume 31MB before the crash. 31MB is no where near 60% of the total on the device even if all I had was the 512MB - http://msdn.microsoft.com/en-us/library/bb331824.aspx – PaulH May 04 '11 at 23:42
Switching to a file-backed implementation, I was able to allocate 26283 times (25MB) before the device froze. I can post that implementation, if necessary. – PaulH May 04 '11 at 23:47
Er, sorry, I was off by an order of magnitude, I thought it was 310 MB. Anyways if you can't get over 31MB out of your 32MB process slot then maybe the answer is... that's why they call it a 32MB process slot? – SoapBox May 05 '11 at 01:04
That's why I've moved to the `MappedFileAllocator` it should allocate memory in the LMA which is outside the 32MB process slot. I can verify that the process slot memory is not filling whereas it does with the `std::allocator`. – PaulH May 05 '11 at 02:20

Memory mapped file std::allocator implementation freezes WM6 device

3 Answers3