I am try to extract url from the href but they will give me the empty list
import requests
from bs4 import BeautifulSoup
headers ={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r =requests.get('https://www.redfin.com/city/5357/WA/Edmonds')
soup=BeautifulSoup(r.content, 'html.parser')
tra=soup.find_all('div',class_='bottomV2')
for links in tra:
for link in links.find_all('a',href=True):
comp=link['href']
print(comp)
CodePudding user response:
Here is the desired output:
import requests
from bs4 import BeautifulSoup
headers ={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r =requests.get('https://www.redfin.com/city/5357/WA/Edmonds',headers=headers)
soup=BeautifulSoup(r.content, 'html.parser')
tra=soup.find_all('div',class_='bottomV2')
for links in tra:
link=links.find('a',href=True)
comp=link['href']
abs_url = f'https://www.redfin.com{comp}'
print(abs_url)
Output:
https://www.redfin.com/WA/Edmonds/31-Pine-St-98020/unit-301/home/21500390
https://www.redfin.com/WA/Lynnwood/18785-76th-Ave-W-98036/home/175276722
https://www.redfin.com/WA/Edmonds/8104-238th-St-SW-98026/unit-C/home/2820302
https://www.redfin.com/WA/Edmonds/6021-145th-St-SW-98026/home/2771575
https://www.redfin.com/WA/Edmonds/21001-88th-Pl-W-98026/home/2652799
https://www.redfin.com/WA/Edmonds/920-Dayton-St-98020/home/2683005
https://www.redfin.com/WA/Edmonds/19730-86th-Pl-W-98026/home/2742187
https://www.redfin.com/WA/Edmonds/1227-8th-Ave-S-98020/home/2757086
https://www.redfin.com/WA/Edmonds/7505-181st-Pl-SW-98026/home/2703399
https://www.redfin.com/WA/Edmonds/1015-Maple-St-98020/home/2682773
https://www.redfin.com/WA/Edmonds/7300-176th-St-SW-98026/home/2718839
https://www.redfin.com/WA/Edmonds/23706-84th-Ave-W-98026/home/2698405
https://www.redfin.com/WA/Edmonds/7217-Meadowdale-Beach-Rd-98026/home/17505861https://www.redfin.com/WA/Edmonds/15419-58th-Pl-W-98026/home/176811540
https://www.redfin.com/WA/Edmonds/15423-58th-Pl-W-98026/home/176811536
https://www.redfin.com/WA/Edmonds/840-Daley-St-98020/home/2683310
https://www.redfin.com/WA/Edmonds/6106-136th-Pl-SW-98026/home/2822544
https://www.redfin.com/WA/Shoreline/115-NW-205th-St-98177/unit-115/home/176726040
https://www.redfin.com/WA/Lynnwood/19814-76th-Ave-W-98036/home/2552538
https://www.redfin.com/WA/Edmonds/15407-58th-Pl-W-98026/home/176686464
https://www.redfin.com/WA/Lynnwood/18751-76th-Ave-W-98037/home/175276737
https://www.redfin.com/WA/Edmonds/7903-218th-St-SW-98026/home/2697000
https://www.redfin.com/WA/Edmonds/907-Dayton-St-98020/home/2682997
https://www.redfin.com/WA/Edmonds/17802-Talbot-Rd-98026/home/2754375
https://www.redfin.com/WA/Edmonds/19126-94th-Ave-W-98020/home/175037005
https://www.redfin.com/WA/Edmonds/191-94th-Ave-W-98020/home/174864268
https://www.redfin.com/WA/Edmonds/12627-Possession-Ln-98026/home/103506228
https://www.redfin.com/WA/Edmonds/827-Fir-St-98020/home/2763817
https://www.redfin.com/WA/Edmonds/9503-Bowdoin-Way-98020/home/2666913
https://www.redfin.com/WA/Picnic-Point-North-Lynnwood/131-Puget-Sound-Blvd-98026/home/108288661
https://www.redfin.com/WA/Edmonds/23726-100th-Ave-W-98020/home/146161823
https://www.redfin.com/WA/Edmonds/7223-224th-St-SW-98026/unit-J9/home/2782416
https://www.redfin.com/WA/Edmonds/0-xxx-Olympic-View-Dr-Unknown/home/175079254https://www.redfin.com/WA/Edmonds/18230-91st-Ave-W-98026/home/2545485
https://www.redfin.com/WA/Edmonds/13531-67th-Ave-W-98026/home/161335515
https://www.redfin.com/WA/Edmonds/15805-72nd-Ave-W-98026/home/2718730
https://www.redfin.com/WA/Edmonds/95-Main-St-98020/home/175589481
https://www.redfin.com/WA/Edmonds/14920-72nd-Ave-W-98026/home/2713344
https://www.redfin.com/WA/Edmonds/7317-Lake-Ballinger-Way-98026/home/2707535
https://www.redfin.com/WA/Edmonds/15604-75th-Pl-W-98026/home/112973708
https://www.redfin.com/WA/Edmonds/Trailside-at-Meadowdale-Beach/Residence-GR-24/home/175129446
CodePudding user response:
Just one alternativ approach, you can use selenium
.
Example
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome('YOUR PATH TO CHROMEDRIVER')
driver.get('https://www.redfin.com/city/5357/WA/Edmonds')
soup=BeautifulSoup(driver.page_source, 'html.parser')
tra=soup.find_all('div',class_='bottomV2')
for links in tra:
for link in links.find_all('a',href=True):
comp=link['href']
print(comp)