Home > database >  Multiple iteration returns same data when scraping multiple urls
Multiple iteration returns same data when scraping multiple urls

Time:06-04

I had a set of urls around 170. All the urls are from same website enter image description here

enter image description here

CodePudding user response:

The two lines below :

# Get 'article_title' and 'article_paragraph' using xpath
article_title = driver.find_element_by_xpath('//h1').text
article_paragraph = driver.find_element_by_xpath("//div[contains(@class,'td-post-content')]").text

should be inside the first loop.

Do everything inside the first loop (even the write to the file). There is no need for the second loop.

  • Related