I wrote about 10 scrapper using undetected chromedriver. They were working all fine until i wanted to create a next one. I started working on it and suddenly i started geting timeouts. When i removed headless, i saw that the cloudflare waiting room that i passed with no struggle kept me in an infinate loop. After that it seems like all of my previous code also broke. Here is a part of the code :
def GetTotalPages(self):
options = ChromeOptions()
options.add_argument('--head')
# options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
with Chrome(options=options) as driver:
driver.get("https://mangasehri.com/manga/?m_orderby=alphabet")
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CLASS_NAME, "pages")))
Lastpage = driver.find_element(By.CLASS_NAME, "pages").text
Lastpage = int(Lastpage.split(" ")[1])
print(Lastpage,"total pages")
return Lastpage
I have tried switching to gecko driver, adding options, nothing seems to be working. Is there an update that has been relased for cloudflare thet blocks it ? Or just my ip got blocked ? if so, how can i get around this issue.