I am new to webscraping and am trying to scrape a website and would like to get all the elements that have data-feature-id=homepage/story
with the ultimate goal of getting a subelement of each of the divs inside of the parent element. I am having trouble getting all of the elements with this parameter and am unsure how to go about doing so.
I've tried the outer div of all the elements that have data-feature-id=homepage/story
; however that results in there being only one element in the list returned.
def getLinks(self):
container = self.driver.find_element(By.XPATH, '//*[@id="__next"]/div[3]/div/main/article/div')
for item in container:
headline = container.find_element(By.XPATH, '//*[@id="__next"]/div[3]/div/main/article/div/div[1]/div/div[1]/a/h3')
print(headline)
The above returns
Type Error:'WebElement' object is not iterable
Can anyone please give some guidance on how to go about this problem?
For your Selenium script, try using the following line instead of your XPath Selector:
driver.find_element(By.CSS_SELECTOR, "div[data-feature-id='homepage/story']")