colly is a web scraping framework written in Go. Import it as https://github.com/gocolly/colly. You will typically use this tag together with the main tag [go].
Questions tagged [go-colly]
63 questions
1
vote
1 answer
go-colly returning empty slice
I am trying to scrape a website, but it seems my slice of products it's empty.
scraper.go:
package scraper
import (
"fmt"
"strings"
"github.com/gocolly/colly"
"github.com/gocolly/colly/extensions"
)
type Product struct {
name …
Antonio Costa
- 101
- 1
- 6
1
vote
1 answer
Go colly returning a string instead of a slice of strings
I am trying to scrape the webpage https://www.brasiltronic.com.br/pesquisa?pg=1&t=Fone%20de%20ouvido
with the following code:
scraper.go
package scraper
import (
"fmt"
"strings"
"github.com/gocolly/colly"
…
Antonio Costa
- 101
- 1
- 6
1
vote
0 answers
Go's go-colly library not returning any data when deployed on heroku
I built a web-scraper using Go's go-colly library and it is working pretty good on my local machine but it is not returning any data when deployed on Heroku. What could be the issue? I deployed it as a docker container on Heroku. I am quite new to…
1
vote
0 answers
Colly (GoLang) Web Scrapper - 403 Forbidden
I am trying to scrape products from mediamarkt site with Colly. Here is my code:
func WebScraper(allowedDomain string, page string, htmlElement string, htmlTag string) {
/*
Order in which Collector's callbacks are executed in:
1.…
Stefan Radonjic
- 1,449
- 4
- 19
- 38
1
vote
1 answer
No responses while scraping website in Go
I'm trying to use Go and Colly to scrape a few details about some listings on Zillow. Here's the script I'm using:
package main
import (
"encoding/csv"
"log"
"os"
"time"
"github.com/gocolly/colly"
…
Hugo Smith
- 11
- 1
1
vote
1 answer
Parsing HTML with go-colly and function returns an empty slice
I'm parsing a web site with the colly framework and something wrong is happening. I have a very basic function getweeks() to grab and return something, yet I'm getting an empty slice instead.
func getWeeks(c *colly.Collector) []string {
var wks…
Bartek R.
- 441
- 1
- 8
- 15
1
vote
1 answer
Getting attribute value with Go Colly
When working with c.OnHTML in "html", how can I get the value of the href attribute inside the #id-card-1 ID?
c.OnHTML("html", func(e *colly.HTMLElement) {
...
linkStr := "#id-card-1[href]" //???
log.Print(e.Attr(linkStr))
...}
This is…
Gabriel
- 5,453
- 14
- 63
- 92
1
vote
1 answer
What is the default mode in GoColly, sync or async?
What is the default mode in which network requests are executed in GoColly? Since we have the Async method in the collector I would assume that the default mode is synchronous.
However, I see no particular difference when I execute these 8 requests…
Jan Bodnar
- 10,969
- 6
- 68
- 77
1
vote
2 answers
I cannot web-scrape forbes top billionares website with colly go
package main
import (
"encoding/csv"
"fmt"
"os"
"github.com/gocolly/colly"
)
func checkError(err error){
if err!=nil{
panic(err)
}
}
func main(){
fName:="data.csv"
file,err:=os.Create(fName)
checkError(err)
defer…
Rahul R Nair
- 39
- 9
1
vote
1 answer
how to scrape attribute in attibute with colly
I try to scrape productId of a product but i can not. please help
html code
1
vote
1 answer
How Can I log-in Amazon with Golang Colly
I am trying login to my amazon buyer account for getting tracking info. I made wordpress-woocommerce login and getting infos but I could not for Amazon.
package main
import (
"fmt"
"log"
"github.com/gocolly/colly"
)
func main() {
…
Melisa
- 310
- 2
- 16
1
vote
1 answer
I cannot print data side by side in JSON array in Golang Colly
I'm taking pictures with golang colly from Amazon and I want to throw these pictures in JSON into a single array (only 1 array for each product images). I scraped the pictures I need, I'm just having a problem with the JSON file. Thank you very much…
Melisa
- 310
- 2
- 16
1
vote
1 answer
How to enable Js on colly
I've had a lot of experiences with Scrapy but or some reasons in this project I should use colly. I'm trying to scrape data from a website but it returns To regain access, please make sure that cookies and JavaScript are enabled before reloading the…
Yuseferi
- 7,931
- 11
- 67
- 103
1
vote
2 answers
go-colly: How can I get HTML title in c.OnResponse, so I can fill the struct?
How can I get HTML.title in c.OnResponse - or is there a better alternative to fill the Struct with url/title/content
At the end I need to fill the below struct and post it to elasticsearch.
type WebPage struct {
Url string `json:"url"`
…
Espresso
- 5,378
- 4
- 35
- 66
0
votes
0 answers
How can I scrape this webpage using Golang? Is it possible with Colly?
I'm attempting to web scrape the data https://www.publix.com/savings/weekly-ad/view-all, but upon opening the webpage you have to click the "Choose a Store" button, select the correct store, then select the button to switch into List View. Would I…
