Powershell webclient trailing dot in URL bug

Question

So, I have a Powershell script that downloads files, e.g.

$mech = new-object System.Net.WebClient
$mech.DownloadFile('http://www.example.com/folder./index.html', 'out.html')
                      Note the trailing dot here ^

which does not work (gives a 404), looking at the actual request sent, it turns out the trailing dot gets taken out which seems to originate from this bug: https://connect.microsoft.com/VisualStudio/feedback/details/386695/

This seems to be fixed in some versions of .Net but Powershell does not use these versions (I'm using PS2.0 in Win7 which uses .Net 2.0.50727.5472, also tried in PS3.0 which uses .Net 4.0.30319.17929, both are not patched note .Net 4.5.something is installed)

So, this might be a simple question but, there is a workaround listed in the above link, how would I apply this in Powershell?

Alternatively, how do I get Powershell to use a patched version of .Net (and which versions are patched?)

I would prefer to not resort to using external programs such as wget otherwise I might as well not write this in Powershell, however if I must, it needs to support UTF8 in the url

Also, the server is not happy with the . replaced with %2e

Nope just tested on a Win8.1 box (I can't get PS4 on Win 7 for some reason, the update is installed but its still V3...) and also doesn't work, I could be wrong but ALL commandlets involving the URIs are parsed by System.Uri which contains the bug. — user3143516, Dec 30 '13 at 09:36

score 0 · Answer 1 · answered Dec 29 '13 at 11:39

0

How about taking the low level approach and using WinInet with P/Invoke? An introduction to P/Invoke is at MSDN. It's a lot more work that using .Net classes though.

Another an alternative could be using a third-party toolkit like cURL. Libcurl has a .Net wrapper, so using it in Powershell should be more easy than the P/Invoke way.

answered Dec 29 '13 at 11:39

vonPryz

22,996
7
54
65

Can't figure out what WinInet is based off, it its written on .Net it will most likely have the same bug, if it's based of something like Internet Explorer, then it should work fine, either way, thanks but thats a bit too in depth for me – user3143516 Dec 31 '13 at 08:54
@user3143516 WinInet is native unmanaged Win32 code. I am fairly sure it doesn't have the same bug, lest you'd encounter similar behavior with, say, IE. – vonPryz Dec 31 '13 at 09:10

score 0 · Answer 2 · answered Dec 31 '13 at 08:51

Well, just in case anyone else needs a solution here is one although I can't exactly say this is an elegant solution but it is one nonetheless. Turns out you can shove C# directly into PS in PS versions >=3.0

$source = @"
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Reflection;
using System.Net;


public class FixedWebClient
{
    public static System.Net.WebClient NewWebClient()
    {
        MethodInfo getSyntax = typeof(UriParser).GetMethod("GetSyntax", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic);
        FieldInfo flagsField = typeof(UriParser).GetField("m_Flags", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
        if (getSyntax != null && flagsField != null)
        {
            foreach (string scheme in new[] { "http", "https" })
            {
                UriParser parser = (UriParser)getSyntax.Invoke(null, new object[] { scheme });
                if (parser != null)
                {
                    int flagsValue = (int)flagsField.GetValue(parser);
                    // Clear the CanonicalizeAsFilePath attribute
                    if ((flagsValue & 0x1000000) != 0)
                        flagsField.SetValue(parser, flagsValue & ~0x1000000);
                }
            }
        }
        return new System.Net.WebClient();
    }
}
"@

#Remove-TypeData 'FixedWebClient'
Add-Type -TypeDefinition $source
$mech = [FixedWebClient]::NewWebClient()
$mech.DownloadFile('http://www.example.com/folder./index.html', 'out.html')

I'm no sure what the scope of the effects are using a method such as this so I create the WebClient object inside it just in case, if someone more experienced in PS/.Net could clarify, that would be great (this is my first PS script, I'd use Perl but seeing as I'm more of a Windows guy, thought I might as well learn some PS).

It would also be nice if someone translates this into pure PS as it would be interesting to see how that works...

You can do that in PS 2.0 too. For more an elegant solution, compile C# as a DLL and add it as a type. Here's [a sample](http://stackoverflow.com/a/20798237/503046). By pre-compiling the DLL, you can be sure it uses intended version of the framework. — vonPryz, Dec 31 '13 at 09:23

Powershell webclient trailing dot in URL bug

2 Answers2