Get-content not producing an array that Invoke-restmethod can process

Question

As a follow-up to this question, instead of using a long array in the script I wanted to draw from a text file. So I replaced this:

$URLs = 'http://websiteone.com','http://websitetwo.com','http://websitethree.com'

with this

$URLs = Get-Content ./urlfile.txt

or (functionally the same as far I know) this

$URLs = @(Get-Content ./urlfile.txt)

But I end up with Invoke-RestMethod : The remote server returned an error: (400) Bad Request.

Incorporating the great response form my last question, my foreach loop looks like this:

foreach($URL in $URLs) {
    $BODY = @([pscustomobject]@{"client" = @{"clientId" = "company"; "clientVersion" = "1.0"}; "threatInfo" = @{"threatTypes" = "MALWARE","SOCIAL_ENGINEERING","THREAT_TYPE_UNSPECIFIED","UNWANTED_SOFTWARE","POTENTIALLY_HARMFUL_APPLICATION"; "platformTypes" = "ANY_PLATFORM"; "threatEntryTypes" = "URL","EXECUTABLE","THREAT_ENTRY_TYPE_UNSPECIFIED"; "threatEntries" = @{"url" = $URL}}})
    $JSONBODY = $BODY | ConvertTo-Json
    $Result = Invoke-RestMethod -Method 'POST' -Uri $Uri -Body $JSONBODY -Headers $HEADERS
    if ( ([string]::IsNullOrEmpty($Result)) ) {} else {write-host $URL "ALERT: Safe browsing match!"}
}

... but this doesn't work if I create the array with the Get-Content cmdlet. If I run the script either way, then type $URLs, I get the exact same data returned. What am I doing wrong with get-content?

FoxDeploy · Accepted Answer · 2020-06-04T19:12:59.870

The Invoke-RestMethod cmdlet is there to make one Rest request at a time and can't take an array.

You will need to add a forEach loop to step through your $urls one at a time, something like this:

foreach($url in $urls){

    $result = Invoke-RestMethod -Uri $url 
    #do something with $result
}

So to integrate into your sample from the previous question, you should have a urls.txt file which looks like this:

http://google.com
http://small.com
https://fast.com/

And then your code would look like this:

$URLs = get-content .\urls.txt
$HEADERS = @{ 'Content-Type' = "application/json" }
$GOOGLE_API_KEY='[API Key]'
$Uri = 'https://safebrowsing.googleapis.com/v4/threatMatches:find?key='+ $GOOGLE_API_KEY

foreach($URL in $URLs) {
    $BODY = @([pscustomobject]@{"client" = @{"clientId" = "company"; "clientVersion" = "1.0"}; "threatInfo" = @{"threatTypes" = "MALWARE","SOCIAL_ENGINEERING","THREAT_TYPE_UNSPECIFIED","UNWANTED_SOFTWARE","POTENTIALLY_HARMFUL_APPLICATION"; "platformTypes" = "ANY_PLATFORM"; "threatEntryTypes" = "URL"; "threatEntries" = @{"url" = $URL}}})
    $JSONBODY = $BODY | ConvertTo-Json
    $result = Invoke-RestMethod -Method 'POST' -Uri $Uri -Body $JSONBODY -Headers $HEADERS

    [pscustomObject]@{SiteName=$url;ThreatInfo=$result.Matches}
}

This would load up the list of $urls from your text file, then run a Rest Request on each, storing the result in $result. Finally, it will make a new PowerShell Object with the site name and show you if there are any matches from the Google SafeBrowsing API.

You'll need to run the command interactively and see which properties from $result are meaningful to you, but you can see all of the expected properties in the Google API Docs.

Edit

Found the bug. It turns out when we use Get-Content the object returned back retains some of the document formatting information from the original file! We can see this by inspecting $JSONBODY. We also see that the conversion to Json from [PSCustomObject is leaving a lot of cruft behind too.

To fix this, we should cast $URL into a string using the ToString() method and also ditch casting to [psCustomObject] too as shown below.

$BODY = @{
        "client" = @{
            "clientId" = "company"; "clientVersion" = "1.0"
            }; 
            "threatInfo" = @{
                "threatTypes" = "MALWARE",
                "SOCIAL_ENGINEERING",
                "THREAT_TYPE_UNSPECIFIED",
                "UNWANTED_SOFTWARE",
                "POTENTIALLY_HARMFUL_APPLICATION"; "platformTypes" = "ANY_PLATFORM"; "threatEntryTypes" = "URL"; "threatEntries" = @{
                    "url" = $URL.ToString()
                }
            }
        }
$JSONBODY = $BODY | ConvertTo-Json

I'm sorry I should have been clear that I was using the foreach loop, I edited the original question to reflect this. — Mullets4All, Jun 04 '20 at 18:52
Check to see if your urls.txt file looks drastically different than what I just editted in. — FoxDeploy, Jun 04 '20 at 18:54

Get-content not producing an array that Invoke-restmethod can process

1 Answers1