how to scrape attribute in attibute with colly

Question

I try to scrape productId of a product but i can not. please help

html code

<span class="info">
 <button data-product="{"merchantName":"xxx","price":"11","productName":"car window","categoryName":"windows","brandName":"aa assosiations","productId":"which I want to scrape"}">

when I try

h.ChildAttr("span.info>button", "data-product")

result is {"merchantName":"xxx","price":"11","productName":"car window","categoryName":"windows","brandName":"aa assosiations","productId":"which I want to scrape"}

and when I try

h.ChildAttr("span.info>button", "productId")

there is no result. how can I get this data with colly?

Never used go-colly before, but perhaps `h.ChildAttr("span.info>button", "data-product.productId")`? — Thijs van der Heijden, Oct 21 '21 at 11:05
thanks but no result again. i think colly not important. goquery or anything for parse html do it with same method, but I can not find from google — Sinan A, Oct 21 '21 at 11:34

Bertrand Martel · Accepted Answer · 2021-10-21T13:39:08.397

0

The attribute value is a raw value, and in this case, it's in JSON format, so you will need to parse the JSON in order to correctly get the data.

For example:

package main

import (
    "log"
    "encoding/json"
    "github.com/gocolly/colly"
)

func main() {
    c := colly.NewCollector()

    c.OnHTML(`body`, func(e *colly.HTMLElement) {
        text := e.ChildAttr("span.info>button", "data-product")

        var result map[string]interface{}
        err := json.Unmarshal([]byte(text), &result)
        if err != nil {
            log.Println(err)
            return
        }
        log.Println(result["productId"])
    })

    c.Visit("[some url]")
}

Output

2021/10/21 14:23:24 which I want to scrape

edited Oct 21 '21 at 13:39

answered Oct 21 '21 at 12:25

Bertrand Martel

42,756
16
135
159

@Shinan I've updated to reflect your changes if that's ok – Bertrand Martel Oct 21 '21 at 13:39
1

thanks for your answer and with your idea `var result map[string]interface{} json.Unmarshal([]byte(body), &result) fmt.Println(result["ProductId"])` – Sinan A Oct 21 '21 at 13:40

how to scrape attribute in attibute with colly

1 Answers1