0

I am using Playwright in F# for web scrapping and I noticed that result is returned randomly. I have this code.

let getContent (url:string) =
task{
    use! paywright = Playwright.CreateAsync()
    let! browser = paywright.Chromium.LaunchAsync()
    printfn "URL  %A" url
    let! page = browser.NewPageAsync()
    page.SetDefaultTimeout(15000f)
    let! goto = page.GotoAsync(url)
    let! price = page.Locator("//span[@class='norm-price ng-binding']").AllInnerTextsAsync()
    
    printfn "Price %A" price
}

When I run the console program sometimes it returns result (list of prices), but sometimes its just finished with empty result. I really dont know what can be wrong. I also try use async wrapper instead of task but the output is same. The delay I increase to 15s, but it also doesnt help.

Dartos Koste
  • 1
  • 1

1 Answers1

1

Could it be that you do not await the task returned by getContent? Maybe the program terminates before writing to the console. If the calling code is not asynchronous (and cannot propagate the task), you could try:

let printContent (url : string) =
    task { ... } |> Task.RunSynchronously

Update 1: Probably the page loads it's price data asynchronously.The default timeout on the page is there to specify a maximum timeout, not to wait that long for some data to arrive in the controlled browser instance. Most likely you'll have to wait for some request to finish or some element to appear on the page. Can you share the URL publicly?

CaringDev
  • 8,391
  • 1
  • 24
  • 43
  • 1
    Actually I have **getContent url |> Task.WaitAll**. Where I call the fucntion. So it shouldnt be that case. – Dartos Koste Sep 28 '22 at 12:36
  • @Asik Do you mean `Task.WhenAll`? `Task.WaitAll` is blocking. Anyways, as you say, I too prefer `Task.RunSynchronously` for waiting for one task. – CaringDev Sep 28 '22 at 14:15
  • Oh, nvm. I'll just delete the comment since it's not true at all. – Asik Sep 28 '22 at 14:18
  • Its actually little bit weird but when I add `let! browser = paywright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions(Headless = false ))` It return a lot more often a results (still sometimes return empty result) – Dartos Koste Sep 29 '22 at 13:38
  • Probably not a answer, but I rewrited it to C# and works well. – Dartos Koste Oct 01 '22 at 14:05