5

I'm following up on this:

Copying large file to remote server causes it to run out of physical memory

But figured I'd post this separately.

I have a 64-bit server, Xeon 5405, with 14GB memory, and a Areca 1680 SATA storage controller, that I use for backups, NMS, and testing. It has Intel Pro/1000EB and PL NICs. I have VMWare server 2.0 running on it, and I want to do some P2V conversions so I can test an application's service pack.

I discovered that in the process of copying the 44GB VMDK file to the server that it runs out of memory. I thought it was a memory leak in VMWare Converter, but doing the conversion on a different 32-bit server running Server 2008 doesn't have the problem, as long as I do the conversion to the 32-bit server's local disk. If I then try to copy the 44GB VMDK file to the server via Windows File Sharing, the server eventually gets all of its memory consumed behind the scenes and the file transfer slows to a crawl.

So I tried FTP from the remote server. Unfortunately the Windows FTP client wants to use a Temp file on C:\ for its work, which won't work since I don't have that much free space on C:. So I used Filezilla as my client, and I see similar high memory usage (although it goes up to 75% instead of 99%), and the transfers start out fast, and then gets really slow. I see this fast/slow/fast/slow behaviour in the FTP client's transfer rate numbers, and similar behaviour in the network

This explains a lot of the issue:

http://blogs.technet.com/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

I tried ESEUtil and doesn't do the memory thing, but its not very fast.

I'm using TeraCopy at this point. It gets 16MB/sec which is pretty crappy but at least its consistent.

I don't think its my storage controller, as I've copied multi-GB files around on the server locally without a problem.

Wondering if folks smarter than I am know the magic answer that makes this go away. I never used to worry about this stuff until 64-bit Windows came along. I also have a problem on the server I'm making a virtual copy of with SQL Server memory usage, but that seems to have stabilized.

Community
  • 1
Brian Spolarich
  • 227
  • 1
  • 4
  • 12
  • Using ESEUTIL to **copy files**?!? Now that's something I wouldn't have imagined in a million years... – Massimo Aug 20 '09 at 19:12

3 Answers3

2

Try using RichCopy 4 by Ken Tamaru.

http://technet.microsoft.com/en-us/magazine/2009.04.utilityspotlight.aspx?pr=blog

Try running with the "Turn off system buffer" option.

HTH.

  • 1
    Robocopy is included by default in Windows 2008, Vista and 7 – BinaryMisfit Aug 20 '09 at 20:26
  • RichCopy doesn't create the memory issue for me regardless of the settings. I got about 10MB/sec copy throughput (it doesn't do a good job of reporting throughput realtime). TeraCopy gave me 18-20MB/sec for the same file over the LAN (Gigabit). Local copy (different disk volumes on same storage controller) gave me 15-19MB/sec according to TeraCopy, so that may be the storage controller is the rate-limiter. – Brian Spolarich Aug 21 '09 at 17:21
0

We had similar out of memory issues copying 30-40 gig files with Windows 2003 32-bit.

Using TeraCopy solved the problem, I get about 55mb/sec over gigabit (upping the buffer to 2meg).

0

Just bumped into this post while looking into a VMware issue but anyways I had this problem with Windows 2008 Standard x86. I found that turning on "Flow Control" on the NIC solved my problem. I was in talks with MS Support about this and they had nothing and I found this fix in a random post!

Hope it gets rid of your headache like it did mine! Good Luck!