How do I transfer 100 GB of xml files over sockets in C#?

Question

We have a machine with tcp socket opened listening to files.

I can connect to the socket in my C# console application using System.Net.Sockets

I want to read data in an xml file (100 Gb) and send it across via socket.

What's the best way? Is there a microsoft.net library that can break it into packets or stream it and send it across?

Do I have to read it chunk by chunk myself?

Please suggest

Sockets is a streaming protocol already. What do you mean by "chunk by chunk"? — Kirk Woll, Aug 28 '12 at 19:28
i didn't know that. do i have to load all of the file contents in an array and call Send method of the socket or is there another way to send data to the client? — InfoLearner, Aug 28 '12 at 19:39
if i set socket type as stream and send 100 gb of data, will it automatically stream it for me and break it into packets? — InfoLearner, Aug 28 '12 at 19:40

score 0 · Answer 1 · answered Aug 28 '12 at 19:31

There is nothing special that you should do, once you open a socket connection, you can keep it open as long as you want. It would be nice to gzip the xml file, transfer it and then use it on the server. I would expect at least 50:1 compression rate for xml files.

Edit: You have to pay attention when you read the file for sending, to use a buffer to read some lines, send, empty buffer, rinse and repeat and do not load the file on client then start sending data through socket connection.

score 0 · Answer 2 · answered Aug 28 '12 at 19:34

This task although may seem quite easy/solvable it has a lot of caveats. The best thing you can do here is use some kind of middleware to do it. A good example of middleware is ICE.

http://www.zeroc.com/doc/Ice-3.3.1/manual/Hello.4.5.html

I'm not sure if it's a good candidate for this particular job, but you'll find something yourself..

score 0 · Answer 3 · answered Aug 28 '12 at 19:35

If you have an HTTP server on the other end (or can put one there) then you might want to look at Windows Background Intelligent Transfer Services

This will hide a lot of complexity around retries, requesting ranges, etc.

There is an article on interacting with the COM API from C# on MSDN Magazine.

score 0 · Answer 4 · edited May 23 '17 at 12:04

0

If the 100Gb is troubling you, you might want to read this for ideas:

Compressed XML

edited May 23 '17 at 12:04

Community

1
1

answered Aug 28 '12 at 19:41

Vman

3,016
2
24
20

How do I transfer 100 GB of xml files over sockets in C#?

4 Answers4