downloading and streaming files with WCF from a remote FTP Server

Question

I'm building a solution where a WCF service acts as a gateway between an FTP server that it must access remotely via FTP protocol (linux server) and a windows client application. the service itself will be hosted on a windows IIS server.

I based my model on an article about streaming files over http using WCF, but the problem is:

I have to wait for the file to download on the windows server first prior straming it to the client and that could be a major performance problem. I want to direct stream the files from the FTP Sever to the client without having to download it first.

here's the code..

public class TransferService : ITransferService{
Starksoft.Net.Ftp.FtpClient ftp = new Starksoft.Net.Ftp.FtpClient();
public RemoteFileInfo DownloadFile(DownloadRequest request)
{
    RemoteFileInfo result = new RemoteFileInfo();
    try
    {
        string filePath = System.IO.Path.Combine(@"C:\UploadFiles\ServerDownloadFiles", request.FileName);
        System.IO.FileInfo fileInfo = new System.IO.FileInfo(filePath);

        ftp = new Starksoft.Net.Ftp.FtpClient("127.0.0.1"); //remote ftp address
        ftp.Open("user", "pass");

        // here is waiting for the file to get downloaded from ftp server
        System.IO.FileStream stream = new System.IO.FileStream(filePath, System.IO.FileMode.Create, System.IO.FileAccess.Write);

        ftp.GetFileAsync(request.FileName, stream,  true);

        stream.Close();
        stream.Dispose();

        // this will read and be streamed to client
        System.IO.FileStream stream2 = new System.IO.FileStream(filePath, System.IO.FileMode.Open, System.IO.FileAccess.Read);

        result.FileName = request.FileName;
        result.Length = stream2.Length;
        result.FileByteStream = stream2;

    }
    catch (Exception ex)
    {

    }
    return result;

 }

The Client like this:

// start service client
            FileTransferClient.TransferServiceClient client = new FileTransferClient.TransferServiceClient();

            LogText("Start");

            // kill target file, if already exists
            string filePath = System.IO.Path.Combine("Download", textBox1.Text);
            if (System.IO.File.Exists(filePath)) System.IO.File.Delete(filePath);

            // get stream from server
            System.IO.Stream inputStream;
            string fileName = textBox1.Text;
            long length = client.DownloadFile(ref fileName, out inputStream);

            // write server stream to disk
            using (System.IO.FileStream writeStream = new System.IO.FileStream(filePath, System.IO.FileMode.CreateNew, System.IO.FileAccess.Write))
            {
                int chunkSize = 2048;
                byte[] buffer = new byte[chunkSize];

                do
                {
                    // read bytes from input stream
                    int bytesRead = inputStream.Read(buffer, 0, chunkSize);
                    if (bytesRead == 0) break;

                    // write bytes to output stream
                    writeStream.Write(buffer, 0, bytesRead);

                    // report progress from time to time
                    progressBar1.Value = (int)(writeStream.Position * 100 / length);
                } while (true);

                // report end of progress
                LogText("Done!");

                writeStream.Close();
            }

            // close service client
            inputStream.Dispose();
            client.Close();

what do you think?

Take 2:

Stream stream;
public Stream GetStream(string filename)
{
    Starksoft.Net.Ftp.FtpClient ftp = new Starksoft.Net.Ftp.FtpClient();
    //string filePath = System.IO.Path.Combine(@"C:\UploadFiles\ServerDownloadFiles", filename);
    //System.IO.FileInfo fileInfo = new System.IO.FileInfo(filePath);

    ftp = new Starksoft.Net.Ftp.FtpClient("127.0.0.1");
    ftp.Open("testuser", "123456");

    stream = new MemoryStream();

    ftp.GetFileAsyncCompleted += new EventHandler<Starksoft.Net.Ftp.GetFileAsyncCompletedEventArgs>(ftp_GetFileAsyncCompleted);
    this.IsBusy = true;

    ftp.GetFileAsync(filename, stream, true);
    return stream;
}

Service Contract:

[ServiceContract]
public interface IStreamingService
{
    [OperationContract]
    Stream GetStream(string filename);

    [OperationContract]
    Boolean GetBusyState();
}

Service Config (Binding):

<basicHttpBinding>
            <binding name="TransferService" maxReceivedMessageSize="2147483647" maxBufferSize="2147483647" transferMode="Streamed">
                <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>
                <security mode="None">
                </security>
            </binding>
        </basicHttpBinding>

Answer 1

Update: The BlockingStream implementation from the article I originally linked was enough to get this working for me.

Service:

public Stream DownloadFile(string remotePath)
{
    // initialize FTP client...

    BlockingStream blockingStream = new BlockingStream();

    // Assign self-removing TransferComplete handler.
    EventHandler<TransferCompleteEventArgs> transferCompleteDelegate = null;
    transferCompleteDelegate = delegate(object sender, TransferCompleteEventArgs e)
    {
        // Indicate to waiting readers that 'end of stream' is reached.
        blockingStream.SetEndOfStream();
        ftp.TransferComplete -= transferCompleteDelegate;
        // Next line may or may not be necessary and/or safe.  Please test thoroughly.
        blockingStream.Close();
        // Also close the ftp client here, if it is a local variable.
    };
    ftp.TransferComplete += transferCompleteDelegate;

    // Returns immediately.  Download is still in progress.
    ftp.GetFileAsync(remotePath, blockingStream);

    return blockingStream;
}

Client:

StreamingService.Service1Client client = new StreamingService.Service1Client("BasicHttpBinding_IService1");
Stream inputStream = client.GetFile(remotePath);
//long length = inputStream.Length; // << not available with streaming

// write server stream to disk 
using (FileStream writeStream = new FileStream(localPath, FileMode.CreateNew, FileAccess.Write))
{
    int chunkSize = 2048;
    byte[] buffer = new byte[chunkSize];
    do
    {
        // read bytes from input stream 
        int bytesRead = inputStream.Read(buffer, 0, chunkSize);

        // etc.  The rest like yours, but without progress reporting b/c length unknown.

Notes:

• I copied the BlockingStream code directly from that article and pasted it into my service project with no modifications.
• I set breakpoints after the lock(_lockForAll) statements in the Read() and Write() methods of BlockingStream, plus a breakpoint in the read loop of the client-side code. I had to use a pretty big file (at least 20x the FTP client's buffer size) to see proof of the streaming. After about 8 straight writes from the FTP client, the service's other thread starting reading from the stream. After a few rounds of that, the service call returned, and the client started reading, too. All three breakpoints were hit alternately, until only the client was catching up, and then finally completed the download.
• I did not use the real Starksoft FTP client in my testing. I wrote a class that reads a file from the local disk asynchronously, using mainly code taken directly from the Starksoft source.
• I also changed the service method signature to match the simplest case of a web method with a streamed response - closer to your 'take 2'. If you can make it work like this, then you should be able to add your other features (MessageContract, file length, etc.) later.
• If your FtpClient is a member of your service class, the TransferComplete event handler should be, as well.
• Make sure you have transferMode=StreamedResponse in your client's binding, otherwise the client will buffer the data even while the service is trying to stream it.
• Please review and test the BlockingStream as carefully as you would anything found on the Internet!

I also came across these in my research, which may be of interest to you:
List of features which can force a streaming method to buffer its response
Question including some suggestions for improving streaming speed
Complete sample application of basic streaming

---

Is that implementation actually streaming the file back to the client? Unless RemoteFileInfo implements IXmlSerializable, I don't think it meets the requirements for a streaming method. From MSDN:

Restrictions on Streamed Transfers

Using the streamed transfer mode causes the run time to enforce additional restrictions.

Operations that occur across a streamed transport can have a contract with at most one input or output parameter. That parameter corresponds to the entire body of the message and must be a Message, a derived type of Stream, or an IXmlSerializable implementation. Having a return value for an operation is equivalent to having an output parameter.

I think your implementation is actually buffering the data three times: once to a file on the server, again to the FileByteStream property of the result, and the third time at the client, before the service call even returns. You could remove two of those buffering delays by enabling streaming on the binding and returning the readable FileStream object directly from your service method. The other properties could be set as headers in the return message. See this answer for an example.

Maybe you can do one better, though. According to the Starksoft doc, in the call to GetFileAsync, "The output stream must be writeable and can be any stream object". It might be possible for you to create an implementation of Stream that allows you to use a single stream object for all purposes. You would create the stream object, pass it directly to the GetFileAsync method, and then return it directly to the client without waiting for the entire file to be downloaded. This might be overkill, but here is an implementation of a blocking read-write stream that you could try.