Accepting cookies with selenium, can't find element-CodePudding

I'm trying to gather some information from transfermarkt with a scraper, but when I open the page with a bot I always get a huge prompt to accept cookies. I've tried finding the element for the accept button by copying the full xpath, copying the relative xpath, and using '//button[@title="ACCEPTEER ALLE"]', but it can never find the element. I've also tried using the wait until method, but that always led to time out errors because it still couldn't find it, and the page loads quickly. Does anyone know what's wrong?

This is the button:

<button title="ACCEPTEER ALLE"  style="padding: 10px 15px; margin: 5px 10px; border-width: 2px; border-color: rgb(92, 166, 255); border-radius: 4px; border-style: solid; font-size: 14px; font-weight: 400; color: rgb(255, 255, 255); font-family: verdana, geneva, sans-serif; width: 225px; background: rgb(0, 25, 63);">ACCEPTEER ALLE</button>

This is my code:

league = "Primera División"
url = "https://www.transfermarkt.nl/schnellsuche/ergebnis/schnellsuche?query={}".format(league)
browser = webdriver.Chrome()
browser.get(url)
sleep(10)
    
button = browser.find_element(By.XPATH, '//button[@title="ACCEPTEER ALLE"]')
button.click()

CodePudding user response：

After checking you inquiry , you can achieve that by doing like so:

 from selenium.webdriver.common.action_chains import ActionChains

 action = ActionChains(driver)
 cookie_page = driver.find_element_by_id('cookie_page_id') 
 action.move_to_element(cookie_page).perform()`

=> Now your inside the cookie page,to find your accept cookies path.

CodePudding user response：

It is inside an iframe, try the below code, its working:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element(By.CSS_SELECTOR, "iframe#sp_message_iframe_575848")))

driver.find_element(By.XPATH,".//*[@title='ACCEPTEER ALLE']").click()

CodePudding user response：

First of all, you have to switch to iframe then click on cookie to accept it.

Full working code as an example:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.add_experimental_option("detach", True)
browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
league = "Primera División"
url = f"https://www.transfermarkt.nl/schnellsuche/ergebnis/schnellsuche?query={league}"#.format(league)

browser.get(url)
browser.maximize_window()
iframe = WebDriverWait(browser, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"#sp_message_iframe_575848")))
cookie = WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.XPATH, '//button[@title="ACCEPTEER ALLE"]'))).click()