13

I have plenty of RAM, however, after starting and finishing a large number of processes, it seems that most of the applications' virtual memory has been paged to disk, and switching to any of the older processes requires a very long time to load the memory back into RAM.

Is there a way, either via Windows API or via kernel call, to get Windows to unpage all (or as much as possible) memory? Maybe by stepping through the list of running processes and get the memory manager to unpage each process's memory?

Kerrek SB
  • 464,522
  • 92
  • 875
  • 1,084

3 Answers3

11

Update 3: I've uploaded my complete program to github.


OK, based on the replies so far, here's a naive suggestion for a tool that tries to get all applications back into physical memory:

  1. Allocate a small chunk of memory X, maybe 4MB. (Should it be non-pageable?)
  2. Iterate over all processes:
    • For each process, copy chunks of its memory to X. (Possibly suspending the process first?)

Suppose you have 2GB of RAM, and only 1GB is actually required by processes. If everything is in physical memory, you'd only copy 256 chunks, not the end of the world. At the end of the day, there's a good chance that all processes are now entirely in the physical memory.

Possible convenience and optimisation options:

  • Check first that the total required space is no more than, say, 50% of the total physical space.
  • Optionally only run on processes owned by the current user, or on a user-specified list.
  • Check first whether each chunk of memory is actually paged to disk or not.

I can iterate over all processes using EnumProcesses(); I'd be grateful for any suggestions how to copy an entire process's memory chunk-wise.


Update: Here is my sample function. It takes the process ID as its argument and copies one byte from each good page of the process. (The second argument is the maximal process memory size, obtainable via GetSystemInfo().)

void UnpageProcessByID(DWORD processID, LPVOID MaximumApplicationAddress, DWORD PageSize)
{
  MEMORY_BASIC_INFORMATION meminfo;
  LPVOID lpMem = NULL;

  // Get a handle to the process.
  HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, processID);

  // Do the work
  if (NULL == hProcess )
  {
    fprintf(stderr, "Could not get process handle, skipping requested process ID %u.\n", processID);
  }
  else
  {
    SIZE_T        nbytes;
    unsigned char buf;

    while (lpMem < MaximumApplicationAddress)
    {
      unsigned int stepsize = PageSize;

      if (!VirtualQueryEx(hProcess, lpMem, &meminfo, sizeof(meminfo)))
      {
        fprintf(stderr, "Error during VirtualQueryEx(), skipping process ID (error code %u, PID %u).\n", GetLastError(), processID);
        break;
      }

      if (meminfo.RegionSize < stepsize) stepsize = meminfo.RegionSize;

      switch(meminfo.State)
      {
      case MEM_COMMIT:
        // This next line should be disabled in the final code
        fprintf(stderr, "Page at 0x%08X: Good, unpaging.\n", lpMem);

        if (0 == ReadProcessMemory(hProcess, lpMem, (LPVOID)&buf, 1, &nbytes))
          fprintf(stderr, "Failed to read one byte from 0x%X, error %u (%u bytes read).\n", lpMem, GetLastError(), nbytes);
        else
          // This next line should be disabled in the final code
          fprintf(stderr, "Read %u byte(s) successfully from 0x%X (byte was: 0x%X).\n", nbytes, lpMem, buf);

        break;
      case MEM_FREE:
        fprintf(stderr, "Page at 0x%08X: Free (unused), skipping.\n", lpMem);
        stepsize = meminfo.RegionSize;
        break;
      case MEM_RESERVE:
        fprintf(stderr, "Page at 0x%08X: Reserved, skipping.\n", lpMem);
        stepsize = meminfo.RegionSize;
        break;
      default:
        fprintf(stderr, "Page at 0x%08X: Unknown state, panic!\n", lpMem);
      }

      //lpMem = (LPVOID)((DWORD)meminfo.BaseAddress + (DWORD)meminfo.RegionSize);
      lpMem += stepsize;
    }
  }

  CloseHandle(hProcess);
}

Question: Does the region by whose size I increment consist of at most one page, or am I missing pages? Should I try to find out the page size as well and only increment by the minimum of region size and page size? Update 2: Page size is only 4kiB! I changed the above code to increment only in 4kiB steps. In the final code we'd get rid of the fprintf's inside the loop.

Kerrek SB
  • 464,522
  • 92
  • 875
  • 1,084
  • 1
    After some testing, I'm finding that this does actually work pretty well! It certainly has the effect that an application thus "unpaged" becomes responsive again and doesn't require any unexpected hard disk access. – Kerrek SB Jun 03 '10 at 17:52
  • 1
    @Tomalak: I made the other answer the accepted one because my program is ultimately based on that idea. It's very useful after a session of mplayer or any Adobe program pages your entire life out to the disk. – Kerrek SB Jun 29 '11 at 23:23
  • Can somebody provide a binary of this tool for download and install? – Tomas Dec 11 '13 at 13:49
  • @Tomas: I have the source code at home, I can upload it to github if you like. I was finding it pretty useful back in its day. (I've since got a new machine with more memory and SSD where the issue is less pressing.) – Kerrek SB Dec 11 '13 at 13:50
  • Kerrek it would be great if you could upload a binary for Windows XP/7! And if it could allow unswap only of processes with given PIDs (e.g. on a command line)... :-) – Tomas Dec 11 '13 at 13:54
  • @Tomas: Yes, that's what it does: It either lists all PIDs, or unpages given PIDs. It's just plain Windows API, so you should be able to compile it for any Windows. I'll get back to you. – Kerrek SB Dec 11 '13 at 13:56
  • Kerrek I am not a Windows developer, don't know anything about compiling under Windows. Binaries would be great! – Tomas Dec 11 '13 at 14:00
  • @Tomas: See edit, I created a [github repository](https://github.com/louisdx/unpage). Includes binaries. – Kerrek SB Dec 12 '13 at 23:41
  • great! You may want to post your answer [here](http://superuser.com/q/600698/93604) and get a bounty! – Tomas Dec 16 '13 at 22:43
  • @Tomas: Interesting. I left a comment there. – Kerrek SB Dec 16 '13 at 23:37
  • @KerrekSB About the errors. I found [this](http://stackoverflow.com/questions/12122323/readprocessmemory-fails-on-some-pages-getlasterror-299#12143764). With my very limited knowledge of c++ i verified that for every failed access the page is indeed protected by the `PAGE_GUARD`-bit. I used `if (meminfo.Protect > 50) { std::cerr << meminfo.Protect << " - " << PAGE_GUARD << std::endl; }` just before the `ReadProcessMemory`-line. `PAGE_GUARD` is 256 and i got 260 with the failed pages. So it is possible to get rid of the errors (maybe even to remove the `PAGE_GUARD` from the pages). – Rik Dec 19 '13 at 15:16
  • Many year later, I have nothing to add but just wanted to thank you, this is exactly what I was looking for and it works great. :) – koedem Apr 09 '22 at 15:20
8

Well, it isn't hard to implement yourself. Use VirtualQueryEx() to discover the virtual addresses used by a process, ReadProcessMemory() to force the pages to get reloaded.

It isn't likely to going to make any difference at all, it will just be your program that takes forever to do its job. The common diagnostic for slow reloading of pages is a fragmented paging file. Common on Windows XP for example when the disk hasn't been defragged in a long time and it was allowed to fill close to capacity frequently. The SysInternals' PageDefrag utility can help fix the problem.

Kevin Panko
  • 8,356
  • 19
  • 50
  • 61
Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
  • Thank you for both those pointers! Let me check if using the two functions makes any difference on the performance of switching processes! – Kerrek SB May 30 '10 at 18:18
  • 2
    Yes, because calling ReadProcessMemory on every byte of every process is surely going to perform better than just paging things back in..... Oh, BTW, you'll need Admin privies and take the SE_DEBUG_NAME privilege in order to use the process memory functions. – Billy ONeal May 30 '10 at 18:37
  • Oh, no, I don't actually want to read every byte, I just want to tell the memory manager that I might be about to do so... Surely the memory manager must know whether parts of a process's memory have been paged out; I'm surprised that it has no interface to request paging memory back for a specific process _without_ actually requesting content from that memory. Re "every byte of every process": I don't actually mind if it does all that in one go while I'm on coffee break, as long as when I come back everything is in memory. – Kerrek SB May 30 '10 at 19:48
  • Since the program I listed below is fundamentally based on this idea, I will make this the accepted answer. I've written out the program into a fairly useful form by now, let me know if you want the full source. I'm finding it a very useful tool! – Kerrek SB Jun 29 '11 at 23:21
  • 1
    @Kerrek SB: `I'm surprised [...] paging memory [...] without actually requesting`; [I'd be surprised the other way around](http://superuser.com/questions/237137). The OS needs to be able to stick to it's job. This is inherently _undemocratic_. That said, [`VirtualLock`](http://msdn.microsoft.com/en-us/library/aa366895(v=vs.85).aspx) will do [close to what you want](http://blogs.msdn.com/b/oldnewthing/archive/2007/11/06/5924058.aspx) with [SetProcessWorkingSetSize](http://msdn.microsoft.com/en-us/library/ms686234.aspx). See [kb108449](http://support.microsoft.com/kb/108449) – sehe Jul 12 '11 at 13:25
  • @sehe: Hm, I'm not sure that VirtualLock is what I want. And about SetProcessWorkingSetSize -- does that actually load everything back into memory if the process has already been paged out to the disk? How do I know the size I have to set? – Kerrek SB Jul 12 '11 at 13:36
  • I think the knowledge base article is about as good as the info gets. Also, note that I'm with [Randolph](http://superuser.com/questions/237137/how-to-control-ram-swap-usage/237186#237186) – sehe Jul 12 '11 at 13:37
  • 1
    Just as a note: The program turned out to be *very* useful, and did exactly what I wanted: I'd run the unpage command after I and my laptop woke up, made coffee, and when I got back, all the "unpaged" processes (browser, mail, Explorer) were *very* responsive. (The explorer alone would be really annoying when paged out, since even opening a right-click context menu would stall the UI for seconds.) – Kerrek SB Dec 11 '13 at 13:54
-2

No, windows provides no such feature natively. Programs such as Cacheman and RAM IDLE accomplish this by simply allocating a large chunk of RAM, forcing other things to page to disk, which effectively accomplishes what you want.

Billy ONeal
  • 104,103
  • 58
  • 317
  • 552
  • 6
    Erm, it sounds as if it would give the opposite of what he wants. –  May 30 '10 at 17:39
  • @Neil: Does he want things paged in or paged out? It's difficult to tell from "unpaged" ... if he wants everything paged in than he is looking for Windows' default behavior, so I assumed he wanted non-default behavior, which would be paging everything out to disk. – Billy ONeal May 30 '10 at 17:42
  • Sorry if that wasn't clear. I want to get everything _into_ physical memory, _off_ the hard disk. Even though I have 2GB RAM, every time I Alt-Tab to another process, it appears that that process has to be retrieved from the disk first, which I want to avoid by making a single "put everything back into physical memory" call, having coffee, and then coming back to a fast and responsive system. – Kerrek SB May 30 '10 at 17:44
  • @Hack59: Then you are looking for default behavior. Windows doesn't just decide randomly to page things to disk -- it does so only when it runs out of physical memory. – Billy ONeal May 30 '10 at 17:48
  • 1
    @Neil: No doubt, but what's the default behaviour for reverting the process? The point is that at some stage I _am_ running more processes than I have physical RAM for, so stuff does get paged to disk, but when most of those processes have finished, I appear to be left with lots of free physical RAM but all my older processes still paged to disk. [to be continued] – Kerrek SB May 30 '10 at 17:57
  • [continued] Maybe I'm wrong about stuff being written to disk; could you recommend a way to check, e.g. using ProcessExplorer? I'm basically going on a subjective feeling; at the end of the day I have four open programs (Acrobat, Opera, Thunderbird, Putty), but switching between any of them requires like five seconds each just reading from disk. – Kerrek SB May 30 '10 at 17:58
  • @Hack59. You can see the number of page faults per process in process explorer. I'd be surprised if that could explain what you are seeing on its own though unless your disc is about to die or something weird like Windows has stopped using DMA mode http://winhlp.com/node/10. – Martin Smith May 30 '10 at 18:05
  • 1
    @MSmith: I don't think it's any problem of that sort; after a fresh reboot everything is perfect. It's only after the machine has been up for a few days and I've run (and finished) lots of video encoding, playback and 3D games that my "constant" processes (browser, email) seem to have been paged to disk and not recovered after the memory becomes available again. Even right now, with about 600MB free RAM, Opera has 70M total page faults, with about 100 new ones per second. Can I find out how much of Opera's memory is on disk, and instruct the system to load it into RAM? – Kerrek SB May 30 '10 at 18:13
  • Basically, in that language, is there a way to modify a running process so that it will never incur any page faults, provided I have enough free physical RAM and do not launch any further processes? Like telling the system, "don't swap to disk any more", without actually disabling swapping? – Kerrek SB May 30 '10 at 18:16
  • @Hack59: No, there's no way to do that other than to disable the page file altogether. – Billy ONeal May 30 '10 at 18:38