Handling "Accept all cookie" popup with selenium when selector is unknown-CodePudding

I have a python script, It look like this.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from webdriver_manager.chrome import ChromeDriverManager 
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from os import path
import time


# Tried this code
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)

browser = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=chrome_options)

links = ["https://www.henleyglobal.com/", "https://markets.ft.com/data"]

for link in links:
    browser.get(link)
    #WebDriverWait(browser, 20).until(EC.url_changes(link))

    #How do I disable/Ignore/remove/escape this "Accept all cookie" popup and then access the website to scrape data?


browser.quit()

So each website in the links array displays an "Accept all cookie" popup after navigating to the site. check the below image.

I have tried many ways nothing works, Check the one after imports

How do I exit/pass/escape this popup and then access the website to scrape data?

CodePudding user response：

If you open your page in a new browser you'll note the page fully loads, then, a moment later your popup appears. The default wait strategy in selenium is just that the page is loaded.

One way to handle this is to simply inspect the page and find the xpath of the popup window. The below code should work for that.

browser.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS)
if link == 'https://www.henleyglobal.com/': 
browser.findElement(By.XPATH("/html/body/div[7]/div/div/div/div[2]/div/div[2]/button[2]")).click()
else:
browser.findElement(By.XPATH("/html/body/div[4]/div/div/div[2]/div[2]/a")).click()

The code is waiting until the element of the pop-up is clickable and then clicking it.

For unknown sites you could try:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-notifications")
webdriver.Chrome(os.path.join(path, 'chromedriver'), chrome_options=chrome_options)

CodePudding user response：

generally, you can not use some universal locator that will match the "Accept cookies" buttons for each and every web site in the world.
Even here, you have 2 different sites and the elements you need to click are totally different on these sites.
For https://www.henleyglobal.com/ site the correct locator may be something like this CSS Selector .confirmation button.primary-btn while for https://markets.ft.com/data site I'd advise to use CSS Selector .o-cookie-message__actions a.o-cookie-message__button.
These 2 elements are totally different: the first one is button while the second is a, they have totally different class names and all other attributes.
You may thing about the Accept text. It seems to be common, so you could use this XPath //*[contains(text(),'Accept')] but even this will not work since on the first page it matches 2 elements while the accept cookies element is the second between them... So, there is no General locators, you will have to define separate locators for each page.
Again, for https://www.henleyglobal.com/ I would prefer

driver.find_element(By.CSS_SELECTOR, ".confirmation button.primary-btn").click()

While for the second page https://markets.ft.com/data I would prefer this

driver.find_element(By.CSS_SELECTOR, ".o-cookie-message__actions a.o-cookie-message__button").click()

Also, generally we always use WebDriverWait expected_conditions explicit waits, so the code will be as following:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

# for the first page
wait.until(EC.element_to_be_clickable((By.XPATH, ".confirmation button.primary-btn"))).click()

# for the second page
wait.until(EC.element_to_be_clickable((By.XPATH, ".o-cookie-message__actions a.o-cookie-message__button"))).click()