How can I download a folder from a webserver with authentication?

Question

I need a script that downloads a certain folder and all its subfolders + files to my pc from a webserver. It needs to be in powershell. I searched a bit and found this:

Invoke-WebRequest http://www.example.com/package.zip -OutFile package.zip

I get this error when I try to run it. But I can't figure out how I can pass the username and password with it. If anyone can help me that would be greatly appreciated! Also how can I specify the folder it should be saved to? Thanks in advance

Why not google for [Invoke-WebRequest](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest)?. You'll find it has a parameter called `Credential` — Theo, Aug 05 '20 at 13:57
@Theo I did google it but I couldn't figure it out, which is why I am asking how to use it — Manuel, Aug 05 '20 at 14:05

score 0 · Answer 1 · answered Aug 05 '20 at 14:02

0

$cred = Get-Credential
Invoke-WebRequest http://www.example.com/package.zip -OutFile package.zip -Credential $cred

answered Aug 05 '20 at 14:02

Andrew Arthur

1,563
2
11
17

I tried that but it still returs the same error. "Authentification required" – Manuel Aug 05 '20 at 14:13
Is this some internal site? If not, tell us what the site is? – postanote Aug 05 '20 at 17:28
@postanote Yes it's an internal site – Manuel Aug 06 '20 at 06:41
Then credentials are just not being passed properly to the site. The code I gave is just a sample, with form field assumptions, since we'd have no idea whether your single document, multiple-document implementation, and or what the from filed - if any are. So, if your site has different fields, then, that's what needs to be used. So, you'll have to spend the time looking at that '$form.' object to find them. – postanote Aug 06 '20 at 07:38
@postanote Thanks for trying to help me, but I guess I cant do this as I have no clue on what to do. Im in a apprenticeship and I never had to do something like this.. I will try something else than invoke-webrequest maybe – Manuel Aug 06 '20 at 07:43
Invoke-WebRequest works just fine, I and many use it daily. Yet, the site you are reaching must provide you with info to pass to. You have to already know that or discover it. Not all sites allow automation, and some change so the previous script that was working may not work in the future. Other sites require differnt auth, SAML, WIA, Certificate, Basic, Digest, you have to know what that is. See the other example I'll post for you. – postanote Aug 06 '20 at 07:55

postanote · Accepted Answer · 2020-08-06T08:06:19.643

Following up regarding my comment.

I ask that question because some sites require you to be specific in the credential presentation vs just one blob of stuff. For example:

$credentials = Get-Credential

$webServerUrl               = 'http://SomeWebSite'
$r                          = Invoke-WebRequest $webServerUrl -SessionVariable my_session
$form                       = $r.Forms[0]
$form.fields['Username']    = $credentials.GetNetworkCredential().UserName
$form.fields['Password']    = $credentials.GetNetworkCredential().Password

$InvokeWebRequestSplat = @{
    Uri        = $($webServerUrl + $form.Action) 
    WebSession = $my_session 
    Method     = 'GET '
    Body       = $form.Fields
}
$r = Invoke-WebRequest @InvokeWebRequestSplat

Update

The follow-up to the comment. This is using IE with PowerShell for site automation.

# Scrape the site to find form data
$url = 'https://pwpush.com'
($FormElements = Invoke-WebRequest -Uri $url -SessionVariable fe)  
($Form = $FormElements.Forms[0]) | Format-List -Force
$Form | Get-Member
$Form.Fields

# Use the info on the site
$IE = New-Object -ComObject "InternetExplorer.Application"

$FormElementsequestURI = "https://pwpush.com"
$Password = "password_payload"
$SubmitButton = "submit"

$IE.Visible = $true
$IE.Silent = $true
$IE.Navigate($FormElementsequestURI)
While ($IE.Busy) {
    Start-Sleep -Milliseconds 100
}

$Doc = $IE.Document
$Doc.getElementsByTagName("input") | ForEach-Object {
    if ($_.id -ne $null){
        if ($_.id.contains($SubmitButton)) {$SubmitButton = $_}
        if ($_.id.contains($Password)) {$Password = $_}
    }
}

$Password.value = "1234"
$SubmitButton.click()

Invoke-WebRequest is Powershell's version of curl. Its alias is even named curl.

SO, in the IVR use case, all you really need to do something like the Facebook and Linkedin examples:

$cred  = Get-Credential
$login = Invoke-WebRequest 'facebook.com/login.php' -SessionVariable 'fb'
$login.Forms[0].Fields.email = $cred.UserName
$login.Forms[0].Fields.pass = $cred.GetNetworkCredential().Password
$mainPage = Invoke-WebRequest $login.Forms[0].Action -WebSession $fb -Body $login -Method Post 

$cred = Get-Credential
$login = Invoke-WebRequest 'https://www.linkedin.com/uas/login?goback=&trk=hb_signin' -SessionVariable 'li'
$login.Forms[0].Fields.email = $cred.UserName
$login.Forms[0].Fields.pass = $cred.GetNetworkCredential().Password
$mainPage = Invoke-WebRequest $login.Forms[0].Action -WebSession $LI -Body $login -Method Post

Yet, notice I on the FB/LI login page, and I'd need to know that even existed before trying this. Note this is old code, That I've not used in a very long while and I don't have a FB account. I passed this on to someone who did.

I still get the same error also with this. It always says "Authentication required" — Manuel, Aug 06 '20 at 07:34

How can I download a folder from a webserver with authentication?

2 Answers2