0

I gave built a webscraper in python using selenium that runs perfectly fine. But I need to run it in headless mode, for a certain reason, however, when I do this I am getting the following error (a bunch of times).

"[1113/144449.454:INFO:CONSOLE(0)] "Access to fetch at 'https://price-api.crypto.com/price/v1/tags' from origin 'https://crypto.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.", source: https://crypto.com/price (0)"

I am scraping data from 50 coin pages like this. https://crypto.com/price/bitcoin

This is my code:

class Scraper():
    
    def __init__(self):

        options = Options()
        options.add_argument("--headless")
        options.add_argument("window-size=1920,1080")
        self.driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options = options)

Then below I have different methods that do things like "accept cookies", "navigate to price page", "get data from coin pages" etc.

The scraper works absolutely fine when I remove "chrome_options = options". Any help appreciated. By the way I am fairly new to python and selenium.

If anyone wants to see the full code: https://github.com/lewis-010/data-collection-pipeline/blob/main/crypto.py

Lewis-010
  • 35
  • 5

1 Answers1

0
Maybe you can set it like this

self.options = webdriver.ChromeOptions()
self.options.add_experimental_option('excludeSwitches', ['enable-automation'])
self.browser = webdriver.Chrome(options=self.options)
Aloha
  • 1
  • 3
  • I still get the same error unfortunately. Do you think if I removed ChromeDriverManager().install() from the self.driver line then it would work? At the minute I need use the ChromDriverManager to allow the file to find chromdriver since I didn't know how to get it in PATH – Lewis-010 Nov 13 '22 at 16:12
  • You can copy the chromedriver to the site_packages folder of the Lib folder in the python folder location – Aloha Nov 14 '22 at 04:40