Home > Software design >  Selenium not finding all the elements
Selenium not finding all the elements

Time:10-11

Selenium can't find all the tags in a webpage. It finds only one, but there are two, and I need those two to make my script work. This is the link. I tried using WebDriverWait, but it still returned me only one of the tags. I tried searching for different divs, but I kept returning me only one. I think it has to do with some of the classes that the second div has, and the first one doesn't, but I am really stuck.

Here's the HTML code.

full_image_link = driver.find_element_by_class_name('sc-1qpw8k9-3')
full_image_link.click()

full_image_link_ = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located(
        (By.CLASS_NAME, 'sc-1qpw8k9-3')
    )
)

CodePudding user response:

You can use the api to retrieve the image links, no need to scrape:

import requests

r = requests.get('https://www.pixiv.net/ajax/illust/93333481/pages?lang=en').json()
image_links = [i['urls']['original'] for i in r['body']]

Result:

['https://i.pximg.net/img-original/img/2021/10/10/23/35/09/93333481_p0.jpg',
 'https://i.pximg.net/img-original/img/2021/10/10/23/35/09/93333481_p1.jpg']
  • Related