Python/Selenium, xpath suddenly stops working-CodePudding

I have a script that extracts names from a crypto list on coinmarketcap. To get the names I use the function:

num = 0
def print_name():
    global num
    num = num   1
    if num == 100:
         exit()
    sleep(0.1)
    names = driver.find_element(By.XPATH, '/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[' str(num) ']/td[3]/div/a/div/div/div/p')
    print(names.text)

To go through the list, there is a number I have to change which is str(num) in the xpath so the xpath moves to the next name in the list every time, but at number 18 the script just randomlly stops working and gives an error:

Message: no such element: Unable to locate element:

even though the xpath stays the same, I checked multiple times, made the driver refresh, even scroll down to maybe load the data but the program just doesn't recognize the path.

It is not because the website blocks the automation, I tried beginning with the number 5, so I make less requests before reaching number 18 but it still didn't work.

Here is the Error Message:

Traceback (most recent call last):
  File "C:\Users\fkahd\PycharmProjects\coingecko api\sandbox.py", line 27, in <module>
    print_name()
  File "C:\Users\fkahd\PycharmProjects\coingecko api\sandbox.py", line 23, in print_name
    names = driver.find_element(By.XPATH, '/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[' str(num) ']/td[3]/div/a/div/div/div/p')
  File "C:\Users\fkahd\PycharmProjects\coingecko api\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1244, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "C:\Users\fkahd\PycharmProjects\coingecko api\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 424, in execute
    self.error_handler.check_response(response)
  File "C:\Users\fkahd\PycharmProjects\coingecko api\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/p"}
  (Session info: chrome=98.0.4758.82)
Stacktrace:
Backtrace:
    Ordinal0 [0x00557AC3 2587331]
    Ordinal0 [0x004EADD1 2141649]
    Ordinal0 [0x003E3BB8 1063864]
    Ordinal0 [0x004101CE 1245646]
    Ordinal0 [0x004103CB 1246155]
    Ordinal0 [0x0043A612 1418770]
    Ordinal0 [0x004286D4 1345236]
    Ordinal0 [0x00438A0A 1411594]
    Ordinal0 [0x004284A6 1344678]
    Ordinal0 [0x004053F6 1201142]
    Ordinal0 [0x004062E6 1204966]
    GetHandleVerifier [0x006FDF22 1680738]
    GetHandleVerifier [0x007B0DBC 2413564]
    GetHandleVerifier [0x005ED151 563089]
    GetHandleVerifier [0x005EBF13 558419]
    Ordinal0 [0x004F081E 2164766]
    Ordinal0 [0x004F5508 2184456]
    Ordinal0 [0x004F5650 2184784]
    Ordinal0 [0x004FF5BC 2225596]
    BaseThreadInitThunk [0x764AFA29 25]
    RtlGetAppContainerNamedObjectPath [0x77BB7A9E 286]
    RtlGetAppContainerNamedObjectPath [0x77BB7A6E 238]

It says: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/div/p"}

But, when I go to inspect element, the path is

/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/div/p

Which is exactly the same as the one in error message.

Html of text: <p color="text3" font-size="1">SCRT</p>

Full code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

driver = webdriver.Chrome("C:\Program Files (x86)\chromedriver.exe")
driver.get("https://coinmarketcap.com/view/defi/")
sleep(2)
driver.refresh()

num = 0
num2 = 17

def loop2():
    global num2
    global names2
    num2 = num2   1
    if num2 == 100:
        exit()
    sleep(0.1)
    names2 = driver.find_element(By.XPATH, '/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[' str(num2) ']/td[3]/div/a/div/div/div/p')

def print_name():
    global num
    num = num   1
    if num == 18:
         loop2()
    sleep(0.1)
    names = driver.find_element(By.XPATH, '/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[' str(num) ']/td[3]/div/a/div/div/div/p')
    print(names.text)

while True:
    print_name()

Quick update here are an example of xpath 17 and 18, but 18 doesn't work:

/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/div/p

/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[17]/td[3]/div/a/div/div/div/p

Thanks for your help, have a nice day.

CodePudding user response：

To extract the names from the Top DeFi Tokens by Market Capitalization within Coinmarketcap you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

driver.get("https://coinmarketcap.com/view/defi/")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "p[class$='coin-item-symbol']")))])

Using XPATH:

driver.get("https://coinmarketcap.com/view/defi/")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//p[contains(@class, 'coin-item-symbol')]")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC