How to avoid verification page when using selenium chrome driver with python?-CodePudding

I want to scrap some data with selenium python. I have this type of screen sometimes :

Do you know to proceed in order to remove this type of verification ? Here my code :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

options = Options()
options.add_argument("start-maximized")
options.add_argument("--disable-web-security")
options.add_argument("--disable-site-isolation-trials")
options.add_argument("--allow-running-insecure-content")


driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

driver.get('THE_WEBSITE_COM')

CodePudding user response：

Selenium specifically and other automation tools have certain user agents and other identifiers which indicate that it's automated. So maybe have a play around with things like that. Some websites use anti bot tools to analyze browsing behaviours and patterns so try to slow it and randomize it eg. random time between page requests

Another trick is to look at the website and try to find if there are any alternative routes to get the information. For example: is there a public API you can use to bypass it? Is there a mobile version of the website? Sometimes mobile versions have less aggressive Captcha enforcement.

CodePudding user response：

What I do most of the time is to launch the browser separately and connect to it using Dev port which is beautifully explained in this article.

To enable Chrome to open a port for remote debugging, we need to launch it with a custom flag –

chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\selenum\ChromeProfile"

Then connect to the browser using this

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
#Change chrome driver path accordingly
chrome_driver = "C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_driver, chrome_options=chrome_options)
print driver.title