Webs scraping links from a business newspaper-CodePudding

I am a social scientist dabbling in web scraping. I have been trying to scrape the links from this Page The links for daily news is arranged within a table structure. This is my code

options = Options()
options.add_argument("--disable-notifications")
driver = webdriver.Chrome(driverLocation,chrome_options=options)

browser = webdriver.Chrome()


url='https://economictimes.indiatimes.com//archive/year-2021,month-1.cms'

browser.get(url)

table = browser.find_element(By.ID,"calender");
rows=table.find_element(By.TAG_NAME, "tr")
td=rows.find_element(By.TAG_NAME, "td")      
print(td.get_attribute('href'))

The get_attribute returns nothing .I have tried a lot of other combinations -by Xpath etc but it's not working for me. Can somebody help me with this?

CodePudding user response：

In case you meant to scrape all the links under td a tag, You can try the below solution.

Code :

browser.maximize_window()
wait = WebDriverWait(browser, 30)
browser.get("https://economictimes.indiatimes.com/archive/year-2021,month-1.cms")

hrefs = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table#calender td a")))

for href in hrefs:
    print(href.get_attribute('href'))

Imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output :

https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44198.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44199.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44200.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44201.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44202.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44203.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44204.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44205.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44206.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44207.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44208.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44209.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44210.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44211.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44212.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44213.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44214.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44215.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44216.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44217.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44218.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44219.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44220.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44221.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44222.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44223.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44224.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44225.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44226.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44227.cms