Windows Server - "Not enough storage available error"

Question

We have two IIS 7 servers that run in web garden. There's a copy of every site on each server. Files are synchronized using DFS (only source files). Images & big files are located on different server using share.

Atleast few time a week, we get error "Not enough storage available" when reading any file or processing a server code. IIS Server is unable to map path to a local disk (not just the one that synchronizes with DFS), there also seems to be a problem loading group policy permissions (we get various errors) and we can't open any share on any server on lan. There are also I/O errors when reading files from other custom applications. Disk speed and latency during this fall-out are both ok. If we try to copy a file or test a drive speed, usually works. Sometimes, we can't even run "my computer" or any other software (for example performance monitor). It can only start if we "run as administrator". There's more than enough storage available on every disk.

Most of the sites stop working (but not all). After server restart, everything works ok for a few days and then the problem reoccurs.

Servers are virtualized using VMWARE, there are 50 IIS pools (some are mem limited) and around 200 running sites on each server. Both servers have 16GB of RAM (only about 80% used). CPU is constantly between 30 - 60%.

We tried changing the IRPStackSize registry property to a higher number, but there were no changes. Problems still exists. We tried to disable antivirus (NOD) – no success.

Any hint? Maybe we are close to IIS limits (to many sites/pools)? Maybe 32bit OS and 16GB of RAM? Maybe there is some 'secret' registry settings... We don't think that everyday restart is long-term solution.

Answer 1

The Not enough storage available error has nothing to do with disk storage. It is about internal memory.

This error is quite easy to reproduce. Simply select a bunch of large files (images or DLL's or whatever), right-click and use the open-with option to open them all with Notepad. This should eat up allot of internal memory. When the swap-file nears it's maximum capacity, random services will start logging Not enough storage available to the eventlog. Often they will stay in a faulted state until they get restarted.

I have come across this issue with my own services, and had to add some extra error handling to force the service process to exit when such an error occurred on a thread. That way the service would restart and recover in an automated manner.

IIS usually handles this situation quite well (compared to an average windows service).

In order to find the process that causes this (usually a memory leak) you should use the performance monitor mmc snap-in, and record the memory usage of running processes (be selective when deciding what to record, since these logfiles can become quite large). Another option would be to use xperf, which is actually more suited for debugging low-level drivers and so, but can be helpful in this scenario as well.

If it turns out to be the w3wc service that's eating up all memory, you could use a tool such as Microsoft's all-new Application Insights (or any other IIS monitoring tool) to figure out which application pool, web-application or website is responsible. Then you can restrict memory usage for that particular pool, or schedule it to recycle gracefully.

IIS can usually cycle a pool in such a way that no one will notice (depending on how the app handles sessions, if it is custom built you might lose sessions when all static variables are flushed). Often a workaround like this is good enough, and expensive fixing of the bug can be avoided (disclaimer: that's not "my style" of solving problems, but most stakeholders love it that way).

Answer 2

32bit Operating system with 16Gb ram will effectively leave 12Gb's unused. The upgrade to 64bit OS should defenitly be considered.

Use of the PAE switch (Physical Adress Extension) could offer a temporary fix.

For info with regards to max memory which a OS can use along with the PAE switch info look at:

http://msdn.microsoft.com/en-us/library/windows/hardware/gg487503.aspx

Regards,

Answer 3

Could be a memory leak? Possible with that number of IIS pools, depending on what they're doing.

Easy to monitor with POOLMON.EXE from Microsoft (Grab it from the official 2003 Support Tools)

Powershell script I run hourly for the duration of 'reboot -> crash'.

$date = get-date -Format ddMMyy-hhmm

.\poolmon.exe -b -n Output-$date.log

#send-mailmessage -smtpserver 10.1.1.11 -to patrick@mail.co.uk -from poolmon@mail.co.uk -subject "Poolmon output $date" -attachments "Output-$date.log"

#Extract top offender

$TAG = (Get-Content Output-$date.log)[3]

$Code = $((($TAG -replace '\s+', ' ').split(" "))[1]) 
$Byte = $((($TAG -replace '\s+', ' ').split(" "))[6])

ECHO "$Code,$Byte" >>chart.csv

Chart.CSV will show you the top TAG and its bytes allocated value.

The Output-$date.log will show you everything.

If you have got a single offender at the top, that stays at the top and increases until crash you have a likely candidate for the cause.

Also look in Eventvwr under system for events from SRV mentioning 'insufficient resources'

Answer 4

First of all, that is a lot of sites to be running with that many IIS worker processes on a 32-bit operating system. As mentioned by other users, a 64-bit operating system would definitely alleviate performance issues.

Secondly, you are receiving errors regarding storage but did not provide any information regarding storage space. Have you checked the drives on all servers to ensure there is adequate storage? Have you checked your virtual memory settings and verified the drive that the page file is on has plenty of space? In VMWare how are the virtual hard drives stored, locally or on shared network storage?

Third, there is no operating system information but being that IIS 7 is running I'm assuming Server 2008. Have you checked to see if there are service packs and hotfixes available for the servers and read the release notes? There may be a fix in one of those.

I would also recommend running PerfMon on all of the servers and looking for long Physical Disk queues, CPU use, Paging File % use, and Memory page faults/sec. Also look at your VMWare monitoring and see if there are any issues with the datastores such as high latency.

This issue could be caused by a number of things including lost communication with storage, page files growing past the limits of the drive or a number of other things. If you could post the exact text of the event in the event viewer and some other diagnostic information that would help. Being that you are having problems with group policies loading and other issues with applications not loading without jumping through some hoops I wouldn't completely rule out a corrupted operating system either at this point either.

Answer 5

This problem occurred for me on a virtual machine. It has a limited space because it's a free bog-standard licence. All the tools are development tools and i need all of them.

The C: drive got full because of successive windows security updates. I couldn't remove anything at all.

Windows thinks it needs disk space to copy files, probably because it's connected to its virtual memory system. I got around the problem in the following way:

1) Cygwin was already installed on the system

2) I cp -rv the file from source to destination

3) Remembered to put quotes around the files

Watch it copy successfully. If you want a job done right, don't use Windows. If you need to install a command line use MobaXterm (it's a smaller install).