I have facing this error even though code runs and gives output, this error part is given especially when it is on .get() or is it?.I have tried solutions of other SSL error code but doesn't seem to work sometimes, gives 44 links as required but sometimes it gives only 10 or 20, it seems inconsistent and what is SSL error code 1, net_error -101 mean?
Also, tried increasing delay in time.sleep() but doesn't seem to help?
In headless mode, it is giving: has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource, did get any solution for this, also? for this tried CORS extension and it also didn't work.
Console output:
[2664:25084:0119/211237.647:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211238.068:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211238.842:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -113
[2664:25084:0119/211238.844:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -113
[2664:25084:0119/211253.591:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211253.879:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211253.959:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211253.963:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -100
[2664:25084:0119/211254.251:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211255.211:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -100
[2664:25084:0119/211255.788:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -100
[2664:25084:0119/211319.752:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211319.973:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.001:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.228:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.285:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.507:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
I am here
[2664:25084:0119/211350.324:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211350.536:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211350.589:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211350.795:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
I am here
Going Down
Going Down
[2664:25084:0119/211420.337:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211420.562:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211420.594:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211420.823:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
Going Down
Going Down
Going Down
[2664:25084:0119/211450.376:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211450.607:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211450.647:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211450.861:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
44
CODE:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import pandas as pd
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re
from dateutil.parser import parse
import scrapy
#To avoid bot detection
options = webdriver.ChromeOptions()
#options.add_argument('--headless')
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("--ignore-certificate-errors")
options.add_argument('--ignore-ssl-errors')
#options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
browser = webdriver.Chrome(ChromeDriverManager().install(),options=options)
#browser.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
#browser.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
#print(browser.execute_script("return navigator.userAgent;"))
##################################################
#options=options)
wait = WebDriverWait(browser, 20)
browser.set_window_size(1280, 800)
#time.sleep(20)
base_url='https://www.legacy.com/obituaries/bostonglobe/obituary-search.aspx?daterange=88888&startdate=20190101&enddate=20210119&townname=Abington&countryid=1&stateid=22&affiliateid=1232'
print('Gttibg url')
browser.get(base_url)
time.sleep(50)
print("I am here")
last_height = browser.execute_script("return document.body.scrollHeight")
time.sleep(5)
print("I am here")
while True:
# Scroll down to the bottom.
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# Wait to load the page.
time.sleep(10)
# Calculate new scroll height and compare with last scroll height.
new_height = browser.execute_script("return document.body.scrollHeight")
time.sleep(5)#***************************
print('Going Down')
if new_height == last_height:
break
last_height = new_height
time.sleep(20)
links=browser.find_elements_by_xpath('/html/body/div[3]/div[2]/div[2]/form/div[3]/div[1]/div[1]/div[3]/div/div[2]/div/div/div[2]/div[1]/div[1]/a')
print(len(links))
for i in links:
print(i.get_attribute('href'))
print(i.text)
How to avoid this error?
