Scraping web with python scrapy and BeautifulSoup?

Asked Jul 30 '21 at 14:28

Active Jul 31 '21 at 13:58

Viewed 39 times

I need to fetch all articles link from a single page and I tried to do this using both libraries python scrapy and beautifulsoup but actually nothing coming out
Scrapy version

import scrapy


class CnnSpider(scrapy.Spider):
    name = "cnn"
    start_url = ['https://edition.cnn.com/business/india/index.html']

    def parse(self, response):
        for links in response.css('span.cd__headline-text'):
            print(links)

BeautifulSoup version

def links():
    url = "https://edition.cnn.com/business/india/index.html"
    page = requests.get(url)
    soup = BeautifulSoup(page.content, "html.parser")
    l = soup.find_all('span',class_="cd__headline-text")
    for element in l :
        print(element)

both function not printing any data and im sure about the element and the css class

edited Jul 31 '21 at 13:58

asked Jul 30 '21 at 14:28

Marco

either Scrapy or requests will not be able to render JavaScript ! even `CNN` website question answered multiple times before. – αԋɱҽԃ αмєяιcαη Jul 30 '21 at 15:03
as a hint for `Scrapy`, DO NOT USE `HTTPS` within `start_url `, just use `http` which will converted later to `HTTPS` if supported. – αԋɱҽԃ αмєяιcαη Jul 30 '21 at 15:04
Yes it answered my question BUT can i relay on this API to build my application ? – Marco Jul 30 '21 at 15:29
I mean i can't see any information about the API provider – Marco Jul 30 '21 at 15:29
I already scraped links using scrapy-seleniumRequest according to your url but post is closed. – Md. Fazlul Hoque Jul 30 '21 at 23:48
do you have it in git ? – Marco Jul 31 '21 at 13:58

Scraping web with python scrapy and BeautifulSoup?

0 Answers0