invoke-webrequest , New-Object -ComObject "InternetExplorer.Application and New-Object net.webclient

Question

i am trying to create task scheduler to get an embed link from a website every hour and i am using powershell to retrieve the info from web.

 $ie = New-Object -ComObject "InternetExplorer.Application"
$ie.navigate("https://servicenow.xxx.com/textsearch.do?sysparm_search=b096b923-92c5-47ad-9a7c-1657847970a6"
while($ie.Busy) {start-sleep -Milliseconds 1000}
$doc = $ie.document.body.getElementsByClassName("formlink") | select href

this can help me to get the info. But this must be in UI. Will failed if run as backend (task scheduler)

$test = Invoke-WebRequest -Uri https://servicenow.xxx.com/textsearch.do?sysparm_search=b096b923-92c5-47ad-9a7c-1657847970a6
$test.ParsedHtml.body 

#or

$test.ParsedHtml.all 

Invoke-webrequest didnot show to piece if info i looking. the string is in outerHTML. The Classname called Formlink only be found in via " New-Object -ComObject "InternetExplorer.Application" .

$webclient = New-Object net.webclient
$webclient.DownloadString("https://servicenow.xxx.com/textsearch.do?sysparm_search=b096b923-92c5-47ad-9a7c-1657847970a6")

Same goes to net.webclient

Why is that so??

the info i need is in outerHTML with New-Object -ComObject "InternetExplorer.Application"

<td title="" class="vt" style="direction: ltr;" colspan="1" 
                           data-original-title=""><a class="linked formlink" href="u_event.do?sys_id=7435d8a0db3b36c020fffd051d961919&amp;sysparm_record_target=incident&amp;sysparm_record_row=1&amp;sy
                           sparm_record_rows=1&amp;sysparm_record_list=123TEXTQUERY321%253Db096b923-92c5-47ad-9a7c-1657847970a6">INC18701854</a></td>

=============================== Update to @Jason Boyd : Tried with each request header, it still does not have the output

User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)



X-P2P-PeerDist: Version=1.1



Accept-Encoding: gzip, deflate, peerdist



Connection: Keep-Alive



Accept: */*



Host: servicenow.xxx.com



X-P2P-PeerDistEx: MinContentInformation=1.0, MaxContentInformation=2.0



Cache-Control: no-cache



Accept-Language: en-US, en; q=0.8, zh-Hans-SG; q=0.5, zh-Hans; q=0.3

And there is one thing, when running

Invoke-WebRequest -Uri https://servicenow.xxx.com/textsearch.do?sysparm_search=b096b923-92c5-47ad-9a7c-1657847970a6 

IE didnt open up the full link, it direct to the homepage. https://servicenow.xxx.com/navpage.do

Notice that it has iframe ( same url all page) in the source code.

What else can i try ??

Answer 1

The only difference between a request sent by IE and a request sent with Invoke-WebRequest would be the request headers. My guess is that the site in question responds differently based on the headers. The most likely suspect, I think, is the user-agent header. I would start by investigating what happens if you change the user-agent header sent by PowerShell to the same value as the user-agent header sent by IE. That command would look something like this:

$YourIEUserAgentHeader = "Put IE's user-agent header here"
Invoke-WebRequest `
    -Uri https://servicenow.dhl.com/textsearch.do?sysparm_search=b096b923-92c5-47ad-9a7c-1657847970a6 `
    -Headers @{ 'user-agent' = $YourIEUserAgentHeader }

If that does not work I would check each of the other headers sent by IE. Your URL, servicenow.dhl.com, does not resolve for me so I cannot test it myself.