I had a set of urls around 170. All the urls are from same website
CodePudding user response:
The two lines below :
# Get 'article_title' and 'article_paragraph' using xpath
article_title = driver.find_element_by_xpath('//h1').text
article_paragraph = driver.find_element_by_xpath("//div[contains(@class,'td-post-content')]").text
should be inside the first loop.
Do everything inside the first loop (even the write to the file). There is no need for the second loop.