NoSuchElementException when loading code using Selenium on Heroku-CodePudding

Error:

ERROR:asyncio:Task exception was never retrieved

2022-03-14T14:08:52.425684 00:00 app[worker.1]: future: <Task finished name='Task-30' coro=<Dispatcher._process_polling_updates() done, defined at /app/.heroku/python/lib/python3.9/site-packages/aiogram/dispatcher/dispatcher.py:407> exception=NoSuchElementException('no such element: Unable to locate element: {"method":"css selector","selector":".media-cards-grid"}\n  (Session info: headless chrome=99.0.4844.51)', None, ['#0 0x5586a25c17d3 <unknown>', '#1 0x5586a231d688 <unknown>', '#2 0x5586a2353c21 <unknown>', '#3 0x5586a2353de1 <unknown>', '#4 0x5586a2386d74 <unknown>', '#5 0x5586a23716dd <unknown>', '#6 0x5586a2384a0c <unknown>', '#7 0x5586a23715a3 <unknown>', '#8 0x5586a2346ddc <unknown>', '#9 0x5586a2347de5 <unknown>', '#10 0x5586a25f249d <unknown>', '#11 0x5586a260b60c <unknown>', '#12 0x5586a25f4205 <unknown>', '#13 0x5586a260bee5 <unknown>', '#14 0x5586a25e8070 <unknown>', '#15 0x5586a2627488 <unknown>', '#16 0x5586a262760c <unknown>', '#17 0x5586a2640c6d <unknown>', '#18 0x7f9745a5d609 <unknown>', ''])>

Here is the problematic part of the code:

    chrome_options = webdriver.ChromeOptions()
    chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--no-sandbox")
    driver = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), chrome_options=chrome_options)
    driver.get(link)
    block = driver.find_element(by=By.CLASS_NAME, value='media-cards-grid')
    all_image = block.find_elements(by=By.CLASS_NAME, value='media-card')
    all_image = all_image[0:10]

    for image in all_image:
        image_link = image.get_attribute('data-src')
        name = image.find_element(By.CLASS_NAME, 'media-card__title').text
        ref = image.get_attribute('href')
        manga = InlineKeyboardMarkup()
        manga.add(InlineKeyboardButton(name, url=ref))
        await bot.send_photo(chat_id=callback.message.chat.id, photo=image_link, reply_markup=manga)
    shwmrrslts = InlineKeyboardMarkup()
    shwmrrslts.add(InlineKeyboardButton('show more results', url=link))
    await callback.message.answer('if you want to see more, click on the button', reply_markup=shwmrrslts)

I searched for a solution on the internet but never found it. also, this code worked fine from my computer:

    driver = webdriver.Firefox()
    driver.get(link)
    block = driver.find_element(By.CLASS_NAME, 'media-cards-grid')
    all_image = block.find_elements(By.CLASS_NAME, 'media-card')
    all_image = all_image[0:10]
    # print(type(all_image))
    for image in all_image:
        image_link = image.get_attribute('data-src')
        name = image.find_element(By.CLASS_NAME, 'media-card__title').text
        ref = image.get_attribute('href')
        manga = InlineKeyboardMarkup()
        manga.add(InlineKeyboardButton(name, url=ref))
        await bot.send_photo(chat_id=callback.message.chat.id, photo=image_link, reply_markup=manga)
    shwmrrslts = InlineKeyboardMarkup()
    shwmrrslts.add(InlineKeyboardButton('show more results', url=link))
    await callback.message.answer('if you want to see more, click on the button', reply_markup=shwmrrslts)

sorry for my bad english, thanks in advance sorry

CodePudding user response：

This error message...

NoSuchElementException('no such element: Unable to locate element: {"method":"css selector","selector":".media-cards-grid"

...implies that the NoSuchElementException was raised when trying to locate the desired element through the following locator strategy:

(by=By.CLASS_NAME, value='media-cards-grid')

You need to construct a more canonical locator strategy which identifies the element uniquely within the DOM Tree

References

You can find a couple of relevant discussions on NoSuchElementException in:

CodePudding user response：

Hello Dan I have been struggling with this today and the problem is that Heroku is slower than your computer to manage Chrome, so you must let Heroku finish their job. To do this was useful to me import:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

With this methods you can transform a find_element function such as

block = driver.find_element(by=By.CLASS_NAME, value='media-cards-grid')

In to

wait = WebDriverWait(driver, 20)
block = wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'media-cards-grid')))

Notice that EC.presence_of_all_elements_located is also developed, just in case you needed. If the problem persists, increase the time of wait, i.e. the 20.

Hope this will solve your problem!