How to bypass Cloudflare bot protection in selenium

Question

I need to grab some information from a site just for education purpose, however i cannot send requests because of the protection. I get The typical Checking-your-browser page shows up first and then i'm being redirected repeatedly. how i can bypass this protection in python selenium?

Answer 1

I had this problem a long time ago and I was able to solve it. Use the code below and enjoy :)

options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(options=options, executable_path=r"webdriver\chromedriver.exe")

///////// EDIT //////////////// this way now is not working !

Answer 2

Use undetected_chromedriver pip package. This is very simple package for fake client.

import undetected_chromedriver


def init_webdriver():
    driver = undetected_chromedriver.Chrome()

    driver.get(url)

    content = driver.page_content

    driver.close()
    driver.quit()

Also you can run it in background

import undetected_chromedriver
from selenium import webdriver



def init_webdriver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless")
    driver = undetected_chromedriver.Chrome(options)

    driver.get(url)

    content = driver.page_content

    driver.close()
    driver.quit()

I tested it 28 Jun 2022. Works very well.

Answer 3

A bit of a late response and no wonder why developers still face this issue repetitively. I am using Java v17 and Gradle v7.4.2. My solution is related to the one explained above but the code is in Java.

@Before
public void setup() {
    WebDriverManager.chromedriver().setup();
    ChromeOptions options = new ChromeOptions();
    // Bypass Cloudflare checks
    options.setExperimentalOption("useAutomationExtension", false);
    options.addArguments("--disable-blink-features=AutomationControlled");
    driver = new ChromeDriver(options);
    driver.manage().window().maximize();
}

Please refer to the Selenium ChromeOptions documentation for further details.

Happy coding.

Answer 4

As March 2022 :

Hi, I had the same problem when using headless Selenium on a Docker Linux image.

I solved it by creating a virtualdisplay right before calling the webdriver:

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 800))  
display.start()

Don't forget to install both pyvirtualdisplay and xvfb: pip install pyvirtualdisplay and sudo apt-get install xvfb

And you must remove the "headless" option in ChromeDriver, here is the complete code I use :

    #Display in order to avoid CloudFare bot detection
    display = Display(visible=0, size=(800, 800))  
    display.start()
  
    options = webdriver.ChromeOptions()
    options.add_argument('--no-sandbox')
    options.add_argument('start-maximized')
    options.add_argument('enable-automation')
    options.add_argument('--disable-infobars')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--disable-browser-side-navigation')
    options.add_argument("--remote-debugging-port=9222")
    # options.add_argument("--headless")
    options.add_argument('--disable-gpu')
    options.add_argument("--log-level=3")
    driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)

Since it was working nicely without headless on my local computer, I figured emulate a real display might do the work aswell. I do not really understand why, but from what I've understood, CloudFare tries to execute javascript code in order to confirm you're not a bot. Having a emulated webpage display helps to do so.

Answer 5

SOLUTION JULY 2021

just add user agent argument in chrome options and set user agent to any value

ops = Options() ua='cat' ops.add_argument('--user-agent=%s' % ua) driver=uc.Chrome(executable_path=r"C:\chromedriver.exe",chrome_options=ops)