3

I have a strange issue that appears to have started recently.

When I copy a large file (Approx 6GB) from my laptop to one of our older fileservers the server runs out of memory after a few seconds.

This has only started recently, maybe since patch Tuesday although I can't be sure.

This server is a Windows 2000 sp4 machine, its a Dell 2950 with 1GB ram (Note: I am sure this server had more than 1GB!, I can't physically check until the end of the day when I can power it down), 3GHz Xeon proc, 4 x 250Gb 7.5k RPM SATA drives in raid 10 and a 1 Gigabit NIC connected to a 1GB port on an intel managed switch.

(Apparently I can't post images so a link will have to do)

Memory usage graph + info during copy

As soon as I stop the copy the memory instantly frees up:

Memory usage graph + info right after copy stopped

I've removed the antivirus which made had no impact. I've changed the "File and Print Sharing for Microsoft Networks" options to balanced.

We have another server, Windows 2000 SP4 with 2GB Ram, 2.8Ghz Intel Quad Core, 6 x 300Gb 15k SAS in raid 10.

When I copy the same 6GB file here the amount of available memory doesn't change.

Is there anything else I can look at while the server is running? As it's in use and not really affected by small file copies I can't reboot it just yet.

Here is a screen shot of some perfmon counters I had open just as the server runs out of memory.

Perfmon counters during copy

Thanks
Gareth

splattne
  • 28,508
  • 20
  • 98
  • 148
Gareth Hastings
  • 191
  • 1
  • 1
  • 8

8 Answers8

2

I'm running into the same problem.

Trying to do a P2V conversion onto a 64-bit server running Windows Server 2008. Any of the normal file transfer methods for the VMDK file (which is 44GB) cause Windows on the destination server to run out of its 14GB RAM after a few minutes due to the file system caching.

Running the P2V conversion or file copy on a 32-bit server doesn't have this problem and memory usage stays reasonable.

Then trying to copy the VMDK file to the destination VMWare server has the same problem.

This page describe exactly what I'm seeing:

http://blogs.technet.com/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

Based on my work this AM ESEUtil seems to be the way to go. It wasn't as fast as I'd expected, but it didn't freak out Windows either.

The Windows FTP client uses a Temp file on C:\ before moving the file to the target destination. Beware! :-)

This is very frustrating.

Brian Spolarich
  • 227
  • 1
  • 4
  • 12
  • Thanks for the link, that explained the underlying problem. I've accepted this as the answer as it seems there isn't actually a fix! Rather a workaround (good job we use exchange too and have access to the ESEUtil!) – Gareth Hastings Sep 02 '09 at 10:21
1

I know this is a bit of a pain but have you tried a 3rd party file copy utility? Windows tends to be kind of dumb/slow about file copies sometimes. Lifehacker did a top 5 list of these utilities, try one of them out and see if you still have the same issue.

http://lifehacker.com/5280976/five-best-alternative-file-copiers

Also, like towo said, check your virtual memory settings. Best practice is that your pagefile should be x1.5 your memory (i.e. 1 GB mem = 1024 MB; 1024*1.5 = 1536 MB page file)

SQLChicken
  • 1,307
  • 8
  • 10
  • I shall try a 3rd party tool next. The page file is set to min 1536, max 3072. It's currently at 1536 – Gareth Hastings Jul 08 '09 at 12:50
  • I tried teracopy and robocopy. Teracopy copies around 3mb/s and I can see the server writing to disk as it's copying. The server doesn't run out of memory, the amount available barely changes. Robocopy says it copies around 20% of the 6GB file in about 20 seconds. Nothing is being written on the server and then it promptly runs out of memory! – Gareth Hastings Jul 08 '09 at 13:17
1

There are known issues with the Network File Copy processes on W2K - if the remote system can't empty the write cache faster than the rate the file data is arriving in over the network then it will steadily consume all physical memory on the server if the file is big enough. Mark Russinovich has some details on the ways this might happen in this article on changes made in Windows Vista's file copy mechanisms. The performance graph you posted looks like this issue and I have seen exactly this sort of behavior in the past where I had a target system with very slow disks and a fast network.

However even though your target OS is a bit old the hardware isn't all that weak and a RAID 10 setup with 4x7.2K SATA drives should be good for somewhere between 60 and 120Meg/sec write speed which is significantly higher than the 39Meg/sec Vista is reporting for your copy. The odd thing here is that if it is a solid, well-configured GigE link then you could hit network transfer rates reaching 70Meg/sec (and maybe a bit higher) for a sustained copy of a large file like this. That said 38Meg/sec isn't abnormal either if there's any other traffic flowing in or out of either the client or server or (as is more likely) that rate is mostly limited by the speed of your local laptop hard drive.

In any case I would check that your RAID 10 was actually healthy - the symptoms here would make me suspect it wasn't able to write as fast as it should be.

Helvick
  • 20,019
  • 4
  • 38
  • 55
0

I've just run a copy on the server itself, local disk to local disk and it's fine. The amount of free memory available doesn't change.

I guess this is more of a networking related issue. I'll check the network card drivers

UPDATE: The drivers are a few years old. I will update them out of hours tonight. Still not sure why this would affect the server suddenly after all this time though!

Gareth Hastings
  • 191
  • 1
  • 1
  • 8
0

Maybe you need to beef up your virtual memory on your system hard drive, where low hard drive space could have caused said problem.

Also, from a logical point of view, the server does not actually need to store a file in memory when copying from file system to file system; it just allocates a buffer in memory the file passes through. Depending on how you copy files, though, some applications will first store the file completely in memory, and then write it to the disk.

Try to use a protocol like FTP - if it still happens, you should probably look into some networking problems.

The interesting question here would be how the server actually stores the files - as you can see, the I/O load is just way down, which means it's not actually writing the file anywhere, just buffering it in memory.

towo
  • 1,887
  • 14
  • 12
  • I tried FTP and got slightly different results. There was a burst of IO until the server ran out of memory, this time the IO tailed off to low value and the memory looked like it was slowly written to disk. When it got back to approx 800mb free mem the IO spiked again and the process was repeated. See graph http://xastrade.co.uk/gdh/ftpcopy.png – Gareth Hastings Jul 08 '09 at 12:48
0

Is this copy operation just via a normal network share (i.e. copying via Windows Explorer or similar)? I experienced a bug like this in under Win2K but that was fixed as far back as SP1 if I remember rightly.

If you are copying via another method then that application/service may be trying to hold the file in RAM until it has it all, in which case you need to upgrade or replace that app/service.

David Spillett
  • 22,754
  • 45
  • 67
0

Are you using Server 2003 64-bit? There is some information on fixing this issue with x64 on Server 2003 here -> http://www.techspot.com/blog/224/slow-system-performance-when-copying-large-files-in-xp-x64-server-2003-x64/

Seems to happen quite a bit in 2003/XP x64 actually. In fact I just realized one of my servers has been experiencing this same issue for a while now and I haven't had time to troubleshoot it.

drgncabe
  • 395
  • 1
  • 6
0

For anyone experiencing this problem, you may want to try using xcopy on the Windows command line (cmd or PowerShell) and turn off file I/O buffering with the /j switch. It's the recommended way to copy very large files. Type "xcopy /?" for more details. (This parameter was added in Windows Server 2008 R2.)

ColdCold
  • 131
  • 3