I got a list (ul) from a website and now I want to loop over the children and their text i.e.
<ul>
<li>
<span>some text</span>
</li>
<li>
<span>some text 2</span>
</li>
<li>
<span>some text 3</span>
</li>
<li>
<span>some text 4</span>
</li>
</ul>
When I print the outcome of the main node, it says ChildNodeCount:4 Children:[]. The childNodeCount is correct, but the children is empty and thus I cannot loop through the children to retrieve the text.
A page has multiple lists, so what I basically want is a list with "UL" elements so I can loop through each UL element, and within that UL element through its LI children.
Anyone knows what I am doing wrong?
chromedp.Nodes(`.product-item__content ul.product-small-specs`, &specs, chromedp.AtLeast(0)),
Also a small side-question. If I have an slice of strings (URL's) and I would like to crawl them one-by-one. How would I do that? Or let me put it this way. If I got to page "A" and I find 20 links, how can I automatically check those links too and if there are links found visit those too?
I tried this code which results in an error:
exception "Uncaught" (1:54): TypeError: this.getClientRects is not a function at Text.text (:2:55)
maxGoroutines := 1
guard := make(chan struct{}, maxGoroutines)
for i := range links {
guard <- struct{}{}
go func(n int) {
retrieveDetails("https://www.bol.com" + links[i].AttributeValue("href"))
time.Sleep(5 * time.Second)
<-guard
}(i)
}
func retrieveDetails(url string) {
opts := append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", false),
)
actx, acancel := chromedp.NewExecAllocator(context.Background(), opts...)
defer acancel()
ctx, cancel := chromedp.NewContext(
actx,
chromedp.WithLogf(log.Printf),
)
defer cancel()
ctx, cancel = context.WithTimeout(ctx, 6000*time.Second)
defer cancel()
var header string
err := chromedp.Run(ctx,
emulation.SetUserAgentOverride("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"),
chromedp.ResetViewport(),
chromedp.Navigate(url),
chromedp.Sleep(1*time.Second),
chromedp.Click("#js-first-screen-accept-all-button"),
chromedp.WaitVisible(`.product-image`),
chromedp.Text("h1", &header, chromedp.AtLeast(0)),
chromedp.Stop(),
)
fmt.Println(header)
if err != nil {
fmt.Println(err)
}
}
