Automating the downloading of data from a website

Question

I am writing a powershell script to extract data from OpenStreetMap automatically for a country. My whole script is written, but not quite functional. My script below:

$ie = New-Object -ComObject InternetExplorer.Application
$ie.Visible = $true
$ie.Navigate("https://www.openstreetmap.org/#map=9/41.5640/21.5173")

do {    
Start-Sleep -Seconds 1
} until($ie.Busy -eq $true -or $ie.ReadyState -ne 4) 

$ie.document.getElementById("query").value = "North Macedonia"
$ie.document.getElementsByName("commit")[0].click()

Start-Sleep -Seconds 1
Start-Sleep -Seconds 1

$selectedcountry = $ie.Document.getElementsByTagName("a") | Where-Object {$_.innerText -eq "North Macedonia"}
$selectedcountry.click()

Start-Sleep -Seconds 1
Start-Sleep -Seconds 1

$download = $ie.Document.getElementsByTagName("a") | Where-Object {$_.innerText -eq "Download XML"}
$download.click()

Sometimes it works and sometimes it doesn't. For example sometimes I get the error:

PS C:\Users\user> $ie.document.getElementById("query").value = "North Macedonia"

You cannot call a method on a null-valued expression.

At line:1 char:1

+ $ie.document.getElementById("query").value = "North Macedonia"

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : InvalidOperation: (:) [], RuntimeException

+ FullyQualifiedErrorId : InvokeMethodOnNull

The script is completely functional if I run it block by block of code. However, that is not the point of a script.

Can someone please explain why this behaviour is happening? If I had to guess, I would say that I would have to implement some kind of an event handler. Also, any other improvements, better practices are welcome. Thanks in advance.

Answer 1

I reckon @Lee_Dailey has the better answer, and what I'd use, but the script is basically telling you there is no "query" object in the page that you're trying to load, for whatever reason. You could try something like:

if (-not $ie.document.getElementById('query')) {
  $ie.document.<get the HTML> | Out-File -Path 'c:\some\path\output.html'
}

I haven't used the InternetExplorer ComObject myself, so you'll have to find out how to get the HTML yourself. Once you have that, you can examine the HTML to find out why it might not have the "query" object on the page.