0

Please help to a C# newbie,

I'm trying to download zipped files from HTTPS site with Script task in SSIS. Each "external" zip file contains "internal" zip file, that contains 3 txt files.

After extensive search, i've enhanced the DownloadFileTaskAsync with "await", ".Wait()" and even "while (webClient.IsBusy) but still manage to only download the empty "external" zip file.

Please help me to find a way to download full "external" file in a way it won't be empty, but will contain the "internal" zip and all 3 txt files inside of it:

    public async void Main()
    {
        WebClient webClient = new WebClient();
        webClient.Credentials = new NetworkCredential("myuser", "password", "https://example.com/Login.htm");

        webClient.DownloadFileTaskAsync(new Uri("https://example.com/#/FROMsender/EXTERNAL_20160706.zip"), @"C:\temp\Test\EXTERNAL_20160706.zip").Wait();
        while (webClient.IsBusy) Thread.Sleep(1000);

        Dts.TaskResult = (int)ScriptResults.Success;
    }

>

Nadya
  • 21
  • 2
  • 1
    What do you mean with empty or not empty ? WebClient will download it, whether the file empty or not. As long as file is there, it will be downloaded. After download you can extract the file. –  Jul 15 '16 at 21:50
  • Well, WebClient only downloads the empty external zip. If I download it manually it has internal zip and txt files. – Nadya Jul 16 '16 at 23:50
  • My concern is that it doesn't reach the file altogether. But I don't know how to even check this. Suggestions? – Nadya Jul 16 '16 at 23:51
  • What you can try probably set the http header like in your webbrowser and try download it again. Some server will reject request if there are no some headers like user-agent. –  Jul 17 '16 at 00:02
  • How do I do that? I'm a total newbie in C# – Nadya Jul 17 '16 at 20:52
  • Take a look at my answer, try it first... –  Jul 17 '16 at 23:17

1 Answers1

0

Try this one, add some headers to webClient:

public async void Main()
{
    WebClientEx webClient = new WebClientEx();  // <= use WebClientEx
    webClient.Credentials = new NetworkCredential("myuser", "password", "https://example.com/Login.htm");

    // Add headers here ------------
    webClient.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36");
    webClient.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    webClient.Headers.Add("Accept-Language", "en-US,en;q=0.5");
    webClient.Headers.Add("Accept-Encoding", "gzip, deflate");
    webClient.Headers.Add("Cache-Control", "max-age=0");
    webClient.Headers.Add("DNT", "1");
    //------------------------------

    webClient.DownloadFileTaskAsync(new Uri("https://example.com/#/FROMsender/EXTERNAL_20160706.zip"), @"C:\temp\Test\EXTERNAL_20160706.zip").Wait();
    while (webClient.IsBusy) Thread.Sleep(1000);

    Dts.TaskResult = (int)ScriptResults.Success;
}

and use this class instead of WebClient:

public class WebClientEx : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
      var webRequest = (HttpWebRequest) base.GetWebRequest(address);
      if (webRequest != null)
      {
        webRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
      }
      return webRequest;
    }
}
  • Thank you. It seems to work, but the downloaded file does not open. I've tried different file sizes, but every file that arrives at '@"C:\temp\Test\EXTERNAL_20160706.zip"' seems to be 5KB and refuses to be extracted. Ideas? – Nadya Jul 18 '16 at 14:33
  • Sometimes the server need specific header, like a cryptical number. You must find it by your own. How about download it using browser ? If that works, then you must find out the request header from browser and set it in your webClient. –  Jul 18 '16 at 14:36
  • Thank you for the help, unfortunately i can't find the number. When i manually download the file, it just pops up a question in the bottom of the browser and asks if i want to save or open the file. When the download is finished, it pops again and asks whether to open the file. I didn't see any change in the header though... I really appreciate your help, but it seems that this site won't allow it any other way but site's way :( – Nadya Jul 19 '16 at 17:30
  • That's nonsense. If you can download it using the browser, then you can do it in code. – Daniel Vygolov Dec 08 '16 at 08:59