Not Clicking and Copying element in Selenium python 3-CodePudding

Hi everyone i am trying to scrape name and phone number from this website but its not clicking and copying the "Click to Show" element required to see phone number. Also after this how can i add multiple (100 ) urls in loop and can i achieve the same with bs4 as it will be faster.

from selenium import webdriver
chrome_path = r"C:\Users\lenovo\Downloads\chromedriver_win32 (5)\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://www.autotrader.ca/a/ram/1500/hamilton/ontario/19_12052335_/?showcpo=ShowCpo&ncse=no&ursrc=pl&urp=2&urm=8&sprx=-2")

driver.find_element_by_xpath('//p[@class="hero-title"]').text
'2011 Ram 1500 Crew Cab Sport'
driver.find_element_by_xpath('//a[@class="link ng-star-inserted"]').click()

selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <a _ngcontent-wfx-c190="" href="javascript:void(0)" class="link ng-star-inserted">...</a> is not clickable at point (1079, 593). Other element would receive the click: <div id="cookie-banner" class="container-fluid cookie-banner" style="display: block;">...</div>

CodePudding user response：

Regarding Click to Show:

You need to close the Cookie setting pop-up and then perform scrollIntoView to click on the Element.

Was able to click on Click to Show with below code:

from selenium import webdriver
import time
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome(executable_path="path to chromedriver.exe")
driver.maximize_window()
driver.implicitly_wait(10)

driver.get("https://www.autotrader.ca/a/ram/1500/hamilton/ontario/19_12052335_/?showcpo=ShowCpo&ncse=no&ursrc=pl&urp=2&urm=8&sprx=-2")

wait =WebDriverWait(driver,30)

wait.until(EC.element_to_be_clickable((By.XPATH,"//button[@class='close-button']"))).click()
option = wait.until(EC.element_to_be_clickable((By.XPATH,"//a[text()= 'Click to show']")))
driver.execute_script("arguments[0].scrollIntoView(true);",option)
option.click()
time.sleep(10)

Regarding doing the same thing with multiple URLs:

You can try like below:

urls = ['url1','url2']
for url in urls:
    driver.get(url)
    ...

If you want this in Beautifulsoup you need to raise this question under Beautifulsoup tag. Right now you have tagged python, selenium and web-scraping.