Can I use Powershell on Windows to extract information from multiple Google searches?

Question

I have a list of several hundred quoted phrases and I would like to just get one number from a Google search for each of these phrases. The number I need is the estimate of the number of search results for each phrase. (For example, the "N" from the line "About N results (Y seconds)"

PS: I ask about Powershell because of a comment on question Use cmd prompt to search a word on google or other search engine where user https://stackoverflow.com/users/1240980/timwagaman implied that it would be easy to do something like this with Powershell. If there is a way to do this without the complications of Powershell, please let me know. I have never use Powershell before.

I'm not sure what you mean about the "complications of PowerShell", but yeah you can make web requests or REST API calls with PowerShell. The relevant cmdlets you want are `Invoke-WebRequest` and `Invoke-RestMethod`. I'm voting to close but you if you get yourself started with powershell and have a specific question about your code, you could get some good answers here. — briantist, Jul 23 '15 at 21:43
@briantist: I was hoping for some example snippets of code. I am not at all familiar with Powershell or the Microsoft's Net framework. How would you suggest I get started? — FrankH, Jul 23 '15 at 21:55
There are approximately 8 bajillion suggestions if you simply google how to learn powershell. — EBGreen, Jul 23 '15 at 22:00
Actually, in keeping with the content of your question and in the interest of accuracy: "About 940,000 results (0.44 seconds) " — EBGreen, Jul 23 '15 at 22:02
@FrankH I don't know your background, but for me I just started doing things in PowerShell that I used to do in batch files, vbscript, or ruby, and figuring out how to do them in powershell instead. If you have no other scripting experience, it could be a bit steeper (just because it's your first, not because it's powershell). Most beginner questions are answered here on SO already, but it's a good place to ask if you run into a snag that isn't covered by an existing question. Just start doing it. — briantist, Jul 23 '15 at 22:17
Possible duplicate of http://stackoverflow.com/questions/27658610/call-google-api-with-powershell — user4317867, Jul 23 '15 at 22:30
@briantist - I'm an old retired programmer, but I mostly had experience with C and unix scripting like awk, sed, etc. No significant experience with web based objects etc. I will try to take your advice. Thanks. — FrankH, Jul 24 '15 at 00:32
@FrankH you'll be fine with powershell. You don't really need much in the way of OO concepts, and the way the shell operates has a lot of similarities to unix scripting (comparison operators such as `-eq`, `-gt`, `-ge`, etc., the concept of piping). Although you have the power of the .Net framework available, most of what you'll use is in the form of "cmdlets" (functions called like shell commands) and piping from one to the other is common. Given your background it will probably be the best shell/programming experience you've had on Windows. — briantist, Jul 24 '15 at 01:02

score 1 · Accepted Answer · answered Jul 24 '15 at 09:35

1

Something to start with:

$userAgent = 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36'
$searchPhrases = @("supercalifragilisticexpialidocious", "david")

$searchPhrases | %{
    $searchLink = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=$_"
    $results = Invoke-WebRequest -URI $searchLink -UserAgent $userAgent
    $json = $results.Content | ConvertFrom-Json

    Write-Host "SearchPhrase: $_ Count: $($json.responseData.cursor.resultCount)"
}

answered Jul 24 '15 at 09:35

David Brabant

41,623
16
83
111

Why use `Invoke-WebRequest` and then manually `ConvertFrom-Json` instead of just using `Invoke-RestMethod` which will covert it for you? – briantist Jul 24 '15 at 15:27
Thanks so much David! That is just what I needed to get started! With your code example and using the web I was able to figure out how to have my search strings in a file (1 string per line) and have the output go to an output file. The biggest question is: Why are the count of results so much smaller than the same request done though Google on Chrome? For a nonsense string that got 48 results on Google directly, this code returned 1. For more reasonable string I got 30K with code, 380K with Google in Chrome. Is it just that the googleapi is different than Google directly? – FrankH Jul 25 '15 at 04:03
@David, Additional questions: (1) The $userAgent is a mystery to me. Setting it to a null string did not seem to change the results I got. Does that make sense? I did not understand what I found on the web about $userAgent . (2) I could only get this code to run by actually pasting it into a "command prompt" where I had first run PowerShell. Is there a way to save the code to a file and execute the file in PowerShell instead? (3) When I do paste the code, I need to type enter twice after the paste to get it to execute and return to the command prompt. Why? – FrankH Jul 25 '15 at 04:24
@FrankH This was just to give a place to start. You might have to explore the google custom search API for better results ( https://developers.google.com/custom-search/json-api/v1/using_rest ). There are quotas on the number of searches you can do through these APIs though. Another option is to do HTML scraping and call the "regular" google search. Much more contrived however. – David Brabant Jul 25 '15 at 08:22
@DavidBrabant Not very contrived with `Invoke-WebRequest` and its HTML parsing actually. – briantist Jul 26 '15 at 02:13

Can I use Powershell on Windows to extract information from multiple Google searches?

1 Answers1