As my title said. To avoid deprecated warning of Selenium.I try to use the new format as I found. and somehow result return only 1 element. How can I fix it to return elements? FYI I scraping website. I'm working on Colab so I can share with my workmates easier.

Here my code

#!pip install selenium
#!pip install IPython
#!pip install pandas
#!apt-get update
#!apt install chromium-chromedriver

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)

import pandas as pd
from IPython.display import display
from selenium.webdriver.common.by import By
website = 'https://www.bitkub.com/fee/cryptocurrency'
driver.get(website)

#pull_data
coins = driver.find_elements(By.TAG_NAME,'tr')

checking what element it contains (which somehow got only 1)

print(coins)
[<selenium.webdriver.remote.webelement.WebElement (session="091bac0ea46529043f4d84786ebc705f", element="76396e66-6516-4dd0-9d85-bc66bfad4676")>]

CodePudding user response：

Ideally coins should have been a list of elements.

However, to extract the name of the coins you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR:

  driver.get('https://www.bitkub.com/fee/cryptocurrency')
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "tbody tr td:nth-child(2) span")))])
  

• Using XPATH:

  driver.get('https://www.bitkub.com/fee/cryptocurrency')
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//following::td[2]//span")))])
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

CodePudding user response：

What is occuring is that you need webdriver waits and wait for visibility for all elements.

wait = WebDriverWait(driver, 10)
trs = wait.until(ec.visibility_of_all_elements_located((By.TAG_NAME, "tr")))

Import:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec