How do you get the response from a 404 page requested from powershell

Question

I have to call an API exposed by TeamCity that will tell me whether a user exists. The API url is this: http://myteamcityserver.com:8080/httpAuth/app/rest/users/monkey

When called from the browser (or fiddler), I get the following back:

Error has occurred during request processing (Not Found).
Error: jetbrains.buildServer.server.rest.errors.NotFoundException: No user can be found by username 'monkey'.
Could not find the entity requested. Check the reference is correct and the user has permissions to access the entity.

I have to call the API using powershell. When I do it I get an exception and I don't see the text above. This is the powershell I use:

try{
    $client = New-Object System.Net.WebClient
    $client.Credentials = New-Object System.Net.NetworkCredential $TeamCityAgentUserName, $TeamCityAgentPassword
    $teamCityUser = $client.DownloadString($url)
    return $teamCityUser
}
catch
{
    $exceptionDetails = $_.Exception
    Write-Host "$exceptionDetails" -foregroundcolor "red"
}

The exception:

System.Management.Automation.MethodInvocationException: Exception calling "DownloadString" with "1" argument(s): "The remote server returned an error: (404) Not Found." ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
   at System.Net.WebClient.DownloadString(Uri address)
   at CallSite.Target(Closure , CallSite , Object , Object )
   --- End of inner exception stack trace ---
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)

I need to be able to check that the page is returned contains the text described above. This way I know whether I should create a new user automatically or not. I could just check for 404, but my fear is that if the API is changed and the call really returns a 404, then I would be none the wiser.

I'm sure this will come in handy: http://stackoverflow.com/questions/23760070/the-remote-server-returned-an-error-401-unauthorized/23761631#23761631 — Raf, Jul 31 '14 at 11:36
@Raf: Invoke-WebRequest makes no difference. Still get an exception that doesn't contain the text I need — CarllDev, Aug 01 '14 at 14:46
The information you are looking for is likely sitting inside the request, not the error. Use `$r = Invoke-WebRequest ..` and then inspect contents of `$r` by `$r | fl ` — Raf, Aug 01 '14 at 14:54
@Raf using $r = Invoke-WebRequest never populates $r because of the exception. BrokenGlass is onto something though. Will post the answer shortly. Thanks for your interest and help! — CarllDev, Aug 01 '14 at 15:11

score 10 · Answer 1 · answered Aug 01 '14 at 03:28

10

Change your catch clause to catch the more specific WebException, then you can use the Response property on it to get the status code:

{
  #...
} 
catch [System.Net.WebException] 
{
    $statusCode = [int]$_.Exception.Response.StatusCode
    $html = $_.Exception.Response.StatusDescription
}

answered Aug 01 '14 at 03:28

BrokenGlass

158,293
28
286
335

I tried this, and though it gives a nice status code, it still doesn't give me the "Could not find the entity requested" text that I'm looking for. Thanks though. – CarllDev Aug 01 '14 at 08:04
1

This is the correct way to get the status code - to get the html response itself you will have to parse the response stream - e.g. see http://stackoverflow.com/questions/7036491/get-webclient-errors-as-string for an example in C# that you can easily adapt to Powershell – BrokenGlass Aug 01 '14 at 13:28
3

Having read the stackoverflow post you pointed me at, I have come to the following code which is exactly what I need: catch [System.Net.WebException] { $exception = $_.Exception $respstream = $exception.Response.GetResponseStream() $sr = new-object System.IO.StreamReader $respstream $result = $sr.ReadToEnd() write-host $result } – CarllDev Aug 01 '14 at 15:12

score 4 · Answer 2 · edited May 23 '17 at 10:28

BrokenGlass gave the answer, but this might help:

try
{
  $URI='http://8bit-museum.de/notfound.htm'
  $HTTP_Request = [System.Net.WebRequest]::Create($URI)
  "check: $URI"
  $HTTP_Response = $HTTP_Request.GetResponse()
  # We then get the HTTP code as an integer.
  $HTTP_Status = [int]$HTTP_Response.StatusCode
} 
catch [System.Net.WebException] 
{
    $statusCode = [int]$_.Exception.Response.StatusCode
    $statusCode
    $html = $_.Exception.Response.StatusDescription
    $html
}
$HTTP_Response.Close()

Response: check: http://8bit-museum.de/notfound.htm 404 Not Found

another approach:

$URI='http://8bit-museum.de/notfound.htm'
try {
  $HttpWebResponse = $null;
  $HttpWebRequest = [System.Net.HttpWebRequest]::Create("$URI");
  $HttpWebResponse = $HttpWebRequest.GetResponse();
  if ($HttpWebResponse) {
    Write-Host -Object $HttpWebResponse.StatusCode.value__;
    Write-Host -Object $HttpWebResponse.GetResponseHeader("X-Detailed-Error");
  }
}
catch {
  $ErrorMessage = $Error[0].Exception.ErrorRecord.Exception.Message;
  $Matched = ($ErrorMessage -match '[0-9]{3}')
  if ($Matched) {
    Write-Host -Object ('HTTP status code was {0} ({1})' -f $HttpStatusCode, $matches.0);
  }
  else {
    Write-Host -Object $ErrorMessage;
  }

  $HttpWebResponse = $Error[0].Exception.InnerException.Response;
  $HttpWebResponse.GetResponseHeader("X-Detailed-Error");
}

if i understand the question then $ErrorMessage = $Error[0].Exception.ErrorRecord.Exception.Message contains the errormessage you are looking for. (source: Error Handling in System.Net.HttpWebRequest::GetResponse() )

score 0 · Answer 3 · answered Sep 01 '18 at 19:20

Another simple example, hope this helps:

BEGIN
{
    # set an object to store results
    $queries = New-Object System.Collections.ArrayList

    Function Test-Website($Site)
    {
        try
        {
            # check the Site param passed in
            $request = Invoke-WebRequest -Uri $Site
        }
        catch [System.Net.WebException] # web exception
        {
            # if a 404
            if([int]$_.Exception.Response.StatusCode -eq 404)
            {
                $request = [PSCustomObject]@{Site=$site;ReturnCode=[int]$_.Exception.Response.StatusCode}
            }
            else
            {
                # set a variable to set a value available to automate with later
                $request = [PSCustomObject]@{Site=$site;ReturnCode='another_thing'}
            }
        }
        catch
        {
            # available to automate with later
            $request = [PSCustomObject]@{Site=$site;ReturnCode='request_failure'}
        }

        # if successful as an invocation and has
        # a StatusCode property
        if($request.StatusCode)
        {
            $siteURI = $Site
            $response = $request.StatusCode
        }
        else
        {
            $response = $request.ReturnCode
        }

        # return the data   
        return [PSCustomObject]@{Site=$Site;Response=$response}
    }
}
PROCESS
{
    # test all the things
    $nullTest = Test-Website -Site 'http://www.Idontexist.meh'
    $nonNullTest = Test-Website -Site 'https://www.stackoverflow.com'
    $404Test = Test-Website -Site 'https://www.stackoverflow.com/thispagedoesnotexist'

    # add all the things to results
    $queries.Add($nullTest) | Out-Null
    $queries.Add($nonNullTest) | Out-Null
    $queries.Add($404Test) | Out-Null

    # show the info
    $queries | Format-Table
}
END{}

Output:

Site                                               Response     
----                                               --------     
http://www.Idontexist.meh                          another_thing
https://www.stackoverflow.com                      200          
https://www.stackoverflow.com/thispagedoesnotexist 404

score -4 · Answer 4 · answered Jul 31 '14 at 19:17

You could try using the Internet Explorer COM object instead. It allows you to check the browser return codes and navigate the HTML object model.

Note: I've found that you need to run this from an elevated PowerShell prompt in order to maintain the COM object definition.

$url = "http://myteamcityserver.com:8080/httpAuth/app/rest/users/monkey"
$ie = New-Object -ComObject InternetExplorer.Application

Add this to See the browser

$ie.visibility = $true

Navigate to the site

$ie.navigate($url)

This will pause the script until the page fully loads

do { start-sleep -Milliseconds 250 } until ($ie.ReadyState -eq 4)

Then verify your URL to make sure it's not an error page

if ($ie.document.url -ne $url) { 
   Write-Host "Site Failed to Load" -ForegroundColor "RED"
} else {
   [Retrieve and Return Data]
}

You can navigate HTML Object model via $ie.document. Using Get-Member and HTML methods such as GetElementsByTagName() or GetElementById().

If credentials are an issue, build this into a function then use Invoke-Command with the -Credentials parameter to define your logon information.

launching IE just for doing a web request sounds like a really bad idea — BrokenGlass, Aug 01 '14 at 02:26

How do you get the response from a 404 page requested from powershell

4 Answers4

Linked