19

It takes about 2 minutes to do a File.Copy(src, dest); for a 1.6GB File from Folder A to Folder B on the same drive. Is there a faster way to do this in C#/.NET in code (w/o hardware) - Something with stream, threading etc?

Would a filestream be quicker? How about a class that chunks up the file using a threadpool, and reads a range of bytes/writes a range of bytes [that sounds like a great way to corrupt the file, but integrity is not priority 1 here, its speed :-)]

I have searched but everyone says use File.Copy, but it's slow (as slow as Windows Copy) - I would prefer not to use a 3rd party tool.

Here are some answers to some of the questions:

Copy time comparison:

> C# : 2.15m  
> Windows Explorer: 2.53m  
> TeraCopy: 2.26m
> FastCopy: 2.24m

Ok, those are not averages and I know they may change slightly on subsequent runs, but I really thought there would be a faster way to copy a file since I assumed that Windows was doing additional security and integrity checks :-(

I'm still hoping for some good answer (like 'oh yea, files over 1.5GB will be x faster if you do buffer m and turn off security n') -- ok, I'm just wishing at this point.

schmoopy
  • 6,419
  • 11
  • 54
  • 89
  • 12
    I think you'd be hard pressed to do a file copy, faster than windows, on a windows box. – Steven Evers Aug 03 '10 at 17:43
  • I assume your drive is a traditional platter-based drive. If so, chunking the file will likely slow things down due to seek times. – Greg Aug 03 '10 at 17:46
  • Copying 1.6GB to a new location on the *same* drive is not an activity that is very nice to the hard drive. It's got to jump around all over the place to support the simulataneous reads and writes. – Kirk Woll Aug 03 '10 at 17:46
  • Tell me, how long does it take you to copy the same file from within Windows Explorer? – Brian S Aug 03 '10 at 17:47
  • SnOrfus - TeraCopy does a much better job than windows does itself. – schmoopy Aug 03 '10 at 17:47
  • Perhaps if you set up a Windows service in the background that will move the file every 10ms or so between the two locations - then it will look to be in two places at once. File moves tend to be pretty quick on the same drive since no data actually is actually copied. (In all seriousness though, you'd probably have to go much lower level than C# to get any gains over Windows. I don't think threading or anything like that is going to yield much of a performance gain because I'd be willing to wager that the speed is caused more by HDD speeds and specs than processor contention). – MisterZimbu Aug 03 '10 at 17:47
  • How long does it takes on other OS whit same Hardware? – Luka Rahne Aug 03 '10 at 17:48
  • 2
    "TeraCopy uses dynamically adjusted buffers to reduce seek times." http://en.wikipedia.org/wiki/Teracopy – dtb Aug 03 '10 at 17:50
  • I answered some of the questions above, thank you all for your input thus far :-) – schmoopy Aug 03 '10 at 17:59
  • If you read up on TeraCopy it doesn't really copy the file at once. It copies the file asynchronously so it only looks like it's faster. The real copy is done in the background. – Matthew Whited Aug 03 '10 at 18:07
  • Did you try to measure RoboCopy in backup mode? That usually is a lot faster than normal mode. If backup mode is faster, indicate so in your question, and other answers might come up. What definitely slows down your copy operation is copying on the same drive. It now needs to do heavy seeking which usually slows down copy operations a lot. What speeds to you get between different drives? – Jeroen Wiert Pluimers Aug 03 '10 at 18:36
  • So... you're asking if you can implement a file copy routine that is faster than the OS's routine. Good luck there. – Ed S. Aug 03 '10 at 20:10
  • https://stackoverflow.com/questions/4743094/how-can-i-move-all-the-files-from-one-folder-to-another-using-the-command-line – pankaj Aug 13 '20 at 17:27

8 Answers8

8

If you are interested in creating a symbolic or hardlink instead of an actual copy then the following Windows APIs might be helpful.

Brian Gideon
  • 47,849
  • 13
  • 107
  • 150
  • 7
    +1 The fastest way to move the mountain is not to move the mountain at all. ;> – dthorpe Aug 03 '10 at 18:21
  • What advantage would SymbolicLink give, i am trying to wrap my head around this and did a sample with P/Invoke - if i try new FileInfo("symbolicFileName.txt").Name it returns the name, but if i try File.Open("symbolicFileName.txt", .....) it says it cannot find the file. The information i found on SymbolicLink is rather useless as far as really explaining what it is and how it might benefit me. What i REALLY want is a file to modify, if its all good, replace the origional, i was doing this with File.Copy both ways. – schmoopy Aug 03 '10 at 19:00
  • 2
    @schmoopy: This answer will not be helpful to you at all in that case. Hardlinks and symbolic links are basically filesystem pointers. So if you attempt to modify a file opened from a hardlink then it will be changing the contents of the original file. Feel free to cast a downvote on my answer. I would but you cannot downvote yourself. It still might be useful to keep the answer otherwise I would just delete it. – Brian Gideon Aug 03 '10 at 19:32
  • I wont downvote, i appreciate your input and that approach could be useful to someone else :-) – schmoopy Aug 03 '10 at 19:37
5

You could try to use the hardware already at your disposal, and allocate a large buffer in memory. By reading and writing in larger chunks you might be able to remove some of the overhead.

However, there isn't much overhead to get rid of, the bottle neck is still the disk I/O. I would expect at best something like a 5% reduction in execution time.

Guffa
  • 687,336
  • 108
  • 737
  • 1,005
5

File.Copy does a call to CreateFile in Kernel32.dll. If you were copying lots and lots of tiny files (think millions) it might be worthwhile to do the P/Invoke to play with the parameters and skip the Permission Demands. One big file 99.999% of the 2 mins is spent inside the driver code.

dtb
  • 213,145
  • 36
  • 401
  • 431
Conrad Frix
  • 51,984
  • 12
  • 96
  • 155
2

I would assume that windows copy, file.copy, Copyfile All use the same underlying operating system calls to do copying. I would doubt that anything you write would outperform the OS internal calls.

rerun
  • 25,014
  • 6
  • 48
  • 78
2

I haven't tried this, but it might be fast.

Do two memory mappings, one to the input file and one to the output, and move memory from one to the other. The page mapping performed by the OS is tuned to the highest possible performance, since it impacts the overall system speed.

Mark Ransom
  • 299,747
  • 42
  • 398
  • 622
  • I thought of this idea as well, with a twist: Do you know if it would be possible to **not** `memcpy` the bytes between the two mapped views, and instead use just a single view, first mapping it to the source, next explicitly demand the whole `VirtualAlloc` range, and then somehow switching the memory-mapped file view to the destination file, keeping that memory intact, which means then finally all you have to do is simply `FlushViewOfFile` the range? – Glenn Slayden Jul 24 '19 at 02:09
  • @GlennSlayden I'm not aware of a way of doing that. If you can come up with one it would be worth leaving your own answer. – Mark Ransom Jul 24 '19 at 03:33
1

For a single file, going to/from the same drive, the answer to your question is: no.

For network transfers, moving tons of files, and other more complex scenarios there may be room for improvement.

If your current requirement is to copy GBs to a second physical (non-symbollic) location on the same disk, then your best chance for improving performance is probably a faster disk.

STW
  • 44,917
  • 17
  • 105
  • 161
0

All will depends of the allocation decided by the OS at the moment of copy, if the file is in cache (do 5x in same way to avoid it).

The most clear result is if your repeat the test multiple times.

alpha_one_x86
  • 50
  • 8
  • check my solution : https://stackoverflow.com/questions/4743094/how-can-i-move-all-the-files-from-one-folder-to-another-using-the-command-line – pankaj Aug 13 '20 at 17:26
0

I think that you should firstly check how much your disk is fragmented. If your file is in many pieces and you don't have large continuous free space for second one then disk must move head much often which is very slow.

Maybe your disk is simple old and slow and you reach it's maximum performance. In that case solution is simple you need to buy new one, or better two and create RAID 0.

You may also check if during copying something else doesn't use the same disk, for example anti-virus or indexing service.

I didn't propose any software solution, because I don't believe there is anything faster than functions provided by OS.

Zuljin
  • 2,612
  • 17
  • 14