I am planning to develop a high performance local data storage.
- Several processes should be able to attach data to that data storage.
- The data vary from basic int values over strings to byte arrays (raw images).
- Other process should be able to access the data using a string identifier.
- The access should be high performance. Copy operations should be avoided as possible.
My thoughts to a solution so far:
- Usa a shared memory with boost::interprocess
- Use a hashmap as super data structure (boost::unordered)
- All processes which want to communicate with that data storage use a dll, which implements functions like
- int getInt(string identifier)
- void putInt(string identifier, int value)
- void* getData(string identifier, int& length)
- void putData(string identifier, void* data, int length)
- Access will be controlled with mutex within the dll.
- Pointers are a problem in shared memory, so boost has the concept of offset_ptr.
What is still questionable for me is the memory management:
- Is it possible to store variable data as a value in a hashmap?
- Is the usage of boost::variant or boost::any possible in combination with boost::interprocess and recommended?
- Or should I define an own structure to store the data.
- How to handle the data which are variable in size (byte arrays, raw images)?
- Should these data be stored in a separte section of the shared memory and only pointers (offset_ptr) should be stored in the hashmap or can these data be stored in the hashmap itself?
- If stored in a separate section, do I have to organize this seperate memory section myself. Is there a concept or a library available to organize a memory section?
- If the hashmap can hold these raw data directly, are pointers to these raw data still valid if the hashmap is rehashed?
Any recommendation are welcome. I am comming from java, so memory management is quite compliacted for me. Thanks in advance.
