Selenium Python - How to manage to get past cookie consent window-CodePudding

I had a working code for a picture scraper. But it stopped working. Tried different solutions I found, but I just don't seem to be able to tell Selenium to accept the cookie consent it redirects to. Any help much appreciated

wd = webdriver.Chrome('D://chromedriver.exe')
wd.execute_script('''window.open("about:blank");''')  
wd.switch_to.window(wd.window_handles[1]) 
wd.get("https://google.com")
wd.implicitly_wait(10)
search_url = "https://www.google.com/search?safe=off&site=&tbm=isch&source=hp&q={q}&oq={q}&gs_l=img"
#wd.find_element(By.XPATH,'//*[@id="uc-btn-accept-banner"]').click()
wd.implicitly_wait(3)
wd.get(search_url.format(q='Volkswagen'))
WebDriverWait(wd,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src^='https://consent.google.com']")))
WebDriverWait(wd,10).until(EC.element_to_be_clickable((By.XPATH,"//div[@id='introAgreeButton']"))).click()

CodePudding user response：

This is against the Google policy. Even if you get around the cookie consent, Most likely you'll be blocked from google for asking too much data.

CodePudding user response：

Change:

WebDriverWait(wd,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src^='https://consent.google.com']")))
WebDriverWait(wd,10).until(EC.element_to_be_clickable((By.XPATH,"//div[@id='introAgreeButton']"))).click()

To:

WebDriverWait(wd, 10).until(EC.presence_of_element_located((By.XPATH, "//*[contains(text(), 'Accept all')]")))
wd.find_elements(By.XPATH, "//*[contains(text(), 'Accept all')]")[1].click()

CodePudding user response：

I suggest you to use google search api. It is free within a certain limit, it will be enough for you.

https://developers.google.com/custom-search/v1/introduction