2

The problem with the current malloc function is that is does not keep track of the variable that stores the returned memory location. As such fragmentation can occur because it is harder to move around memory.

The MMU can solve this only to a certain extent. Lets say that instead malloc took a double pointer and kept track of the variable. Calls to free would allow free to move around memory and change the memory location.

It is highly unlikely that I am the first to think about this so I am wondering if there is a standard C function that does this or POSIX function?

I understand that this idea is not perfect. The program would have to pass around the same variable instead of copying it however it does solve the issue of fragmentation which does matter to me as I work with low memory devices.

Mahonri Moriancumer
  • 5,993
  • 2
  • 18
  • 28
user3462295
  • 290
  • 2
  • 9
  • 1
    You don't really need a double pointer, you just need an opaque handle that you do not treat like a traditional pointer. But then you're not writing in C anymore. – Cody Gray - on strike Jun 18 '14 at 04:16
  • Note that such a pointer could not be used by any function that was not expecting it , because everybody using this pointer has to now do a double-indirection. This is certainly possible, but it's not built in to Standard C. Addons to C++ for garbage collection do do this. – M.M Jun 18 '14 at 04:28
  • its not a problem, its a benefit -- Less overhead. This is C. You are free to implement whatever memory allocation scheme you want. In the vast majority of cases, the overhead of garbage collection is not desired by c programmers. Personally, to eliminate fragmentation, I like pools of memory. When you are done with a transaction, you just dump the current pool. If you use a circular buffer for your pools, you never fragment memory. – johnnycrash Jun 18 '14 at 05:15
  • @Matt McNabb well yes that is true. However by making a custom allocator the program has control of when to move stuff around is needed. If the OS was managing this however and could interrupt the programs running to move stuff around whenever it feels like then your concern would be valid. Functions would need to be rewritten only if pointers get moved around when the program did not want this to happen. – user3462295 Jun 18 '14 at 15:11

4 Answers4

3

Of course, the realloc() function will, if necessary, move a previously allocated block of memory to a new (larger) location. However, it does not necessarily impact fragmentation.

My solution (in C) has been (in low-memory conditions) to do my own memory management.

Mahonri Moriancumer
  • 5,993
  • 2
  • 18
  • 28
  • Realloc only updates the one pointer. Here is why that is problematic: Lets say that we have three different blocks of memory that just happen to be contiguous. I know that most implementations seem to leave some space but lets pretend that does not happen. We want to resize the block of memory that is in the middle. Because we cannot move around block zero or block two we have to put block one after block two or before block zero. Also I have implemented my own memory management when appropriate. I just thought that there is a standard function that I could use to simplify code. – user3462295 Jun 18 '14 at 04:55
  • @user3462295, correct. I have worked in environments which have the memory allocation constructs you seek; but they have been far from POSIX, and very proprietary. For example, NetWare OS APIs offered allocation techniques that would require the address of a pointer to which transitory memory was allocated. In such a case, the memory could only be referenced using the pointer by which the memory was allocated. The actual address of the allocated memory was changed at the whim of the OS; which moved the allocation in order to reduce memory fragmentation. – Mahonri Moriancumer Jun 18 '14 at 05:02
  • Unfortunately, there is nothing in standard C, POSIX, etc., which comes close. Hence, we must often resort to our own memory management methods. – Mahonri Moriancumer Jun 18 '14 at 05:04
  • I kind of figured that it would turn out to be like this however I just wanted to be sure. – user3462295 Jun 18 '14 at 05:15
1

Though not the same mechanism, the closest thing to what you are speaking of are smart pointers as implemented with the boost libraries. However, they are built for C++.

Smart pointers are 'smart' in that they don't hang around after they aren't needed (and you don't have to free them) so they avoid most of the fragmentation problems you cite.

Jiminion
  • 5,080
  • 1
  • 31
  • 54
  • How do these "smart pointers" you speak of "know" when you are done with it? Also there is no mention that they are free of fragmentation according to this page http://www.boost.org/doc/libs/1_55_0/libs/smart_ptr/smart_ptr.htm I have not used boost. I like low level programming. I will have to look at the source code for this but I do think that manually freeing is better as overhead may be introduced by checking if the program is done with the allocated memory or not. – user3462295 Jun 18 '14 at 05:10
  • smart pointers will not garbage collect circular references like child nodes referring to their parents. – johnnycrash Jun 18 '14 at 05:21
  • They know when to delete themselves because of scope. No, the overhead on them is minimal. OTOH, moving around memory chunks to avoid fragmentation might NOT be minimal. – Jiminion Jun 18 '14 at 13:11
1

Keeping track of which variable points to which memory location cannot be done by just passing a pointer to a pointer. What if the address of the allocated memory is copied to another variable? C is unlike Java that keeps track of object references. In your case, you may be better off managing memory on your own by preallocating a large chunk of memory and splitting it as needed, keeping track of usage, in brief, implementing your own memory management.

Tarik
  • 10,810
  • 2
  • 26
  • 40
  • Well yes you need to store the pointer to the pointer in ram. Also that is the disadvantage to this method you need to pass around that pointer instead of copying it. Also yes just write my own manager using methods like this. It works for my needs. I was just wondering if there is a standard function that does this. – user3462295 Jun 18 '14 at 14:47
0

The idea is not that attractive when you start thinking about implementation details. You can of course write a function that returns a double pointer. What about the intermediate pointer? Where should it be stored? The program itself clearly cannot store it, because the intermediate oointer should have exactly the same lifetime as the pointed-to memory. So the allocator itself should store and manage it. And the memory where the intermediate pointer lives is not movable. But the killer misfeature is that the scheme is not thread safe. As pointers are free to change under the hood, each pointer dereference now requires a lock.

n. m. could be an AI
  • 112,515
  • 14
  • 128
  • 243
  • I usually work with embedded low ram applications that only have one cpu. Yes the allocator does store the pointer to the pointer. Also changes to where memory is located would not occur that often only when memory is allocated or free'd or resized. I do not expect losing a huge amount of performance due to use locks. The OS would manage this. My idea is that lets say you have two programs running. The first allocates some memory and it turns out there is no other way but to move around memory. The OS can simply interrupt the second for a little bit to do the necessary moving. – user3462295 Jun 18 '14 at 14:57
  • Now of course giving the OS the ability to move pointers around whenever it wants is a terrible idea. I was just explaining that the thread safety concerns are minimal. My idea is that is the allocation function fails the program can call a function that moves memory around and then the allocation can be retried. I am not saying always move memory around whenever possible. I mean do it only to prevent the system from crashing. – user3462295 Jun 18 '14 at 15:06
  • It doesn't really matter which part of your system moves memory around. It could be your OS or the program itself. It doesn't also matter how often this happens, or how many CPUs you have. If it can happen at all, your threads are not safe. I'm not sure how thread safety could be quantified (more vs less concern). It's either safe or not. You are welcome to try and write such an allocator yourself and see how it behaves. – n. m. could be an AI Jun 18 '14 at 15:16
  • Yes that is true. In regards to thread safety what I meant is that making it thread safe would not be a huge challenge it appears as though my previous comment was ambiguous in that regard. Also I take back my "It would be terrible" to have the OS move memory whenever it wants statement. It would just involve code changes to account for this. Also I have written an allocator with a similar concept before however I think I will take up "You are welcome to try and write such an allocator yourself and see how it behaves." I will post the code when I do such. – user3462295 Jun 18 '14 at 15:23
  • I have written an allocator that uses a similar idea but is not exactly the same here: https://github.com/ComputerNerd/Open-Jazz-Jackrabbit-Casio-Prizm-port/blob/master/src/mem.cpp and the header is here https://github.com/ComputerNerd/Open-Jazz-Jackrabbit-Casio-Prizm-port/blob/master/src/mem.h this is a very basic implantation of my idea. I wrote it a while ago. However I plan to write a better one. – user3462295 Jun 18 '14 at 15:32
  • "making it thread safe would not be a huge challenge it appears as". I serioudly doubt it. IME problems of thread safety are normally more challenging than they appear at first, not less. – n. m. could be an AI Jun 18 '14 at 15:50