I'm new to Python and web scraping. I was trying to extract data from Rentals.ca. My code is below and I couldn't retrieve any data from the website. I have tried other websites and successfully get the data. I have tried to follow some Youtube tutorials, but they didn't solve my problems.
from csv import writer
from bs4 import BeautifulSoup
import urllib.request
with urllib.request.urlopen("https://rentals.ca/new-westminster?bbox=-123.20298,49.13647,-122.75322,49.31987") as url:
Soup = BeautifulSoup(url, 'html.parser')
lists = Soup.find_all('div', class_="container")
with open('testing.csv', 'w', encoding='utf8', newline='') as f:
thewriter = writer(f)
for list in lists:
Price = list.find('p', class_="listing-card_price").text
Type = list.find('div', class_="listing-card__type-and-updated").text
Title = list.find('div', class_="listing-card").text
print(Price, Type, Title)
```
CodePudding user response：
The following is one way of obtaining that data - inspect browser's Dev tools - Network tab for any XHR calls, and then scrape that eventual API endpoint. Here is the code:
import requests
import pandas as pd
url = 'https://rentals.ca/phoenix/api/v1.0.2/listings?details=mid1&suppress-pagination=1&limit=350&bbox=-123.20298,49.13647,-122.75322,49.31987'
headers = {
'accept-language': 'en-US,en;q=0.9',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'
}
s = requests.Session()
s.headers.update(headers)
s.get('https://rentals.ca/new-westminster?bbox=-123.20298,49.13647,-122.75322,49.31987')
r = s.get(url)
df = pd.json_normalize(r.json()['data']['listings'])
print(df)
You get a quite comprehensive dataframe with the listings:
id name address1 address2 postal_code phone url view_on_map_url city format title featured_status photo_count raw_property_type property_type updated rent_range beds_range baths_range dimensions_range location.lat location.lng location.x location.y photo.id photo.url photo.width photo.height photo.priority photo.unit_id photo.scales.large.url photo.scales.large.width photo.scales.large.height photo.scales.medium.url photo.scales.medium.width photo.scales.medium.height photo.scales.small.url photo.scales.small.width photo.scales.small.height
0 612435 None 1471 Hunter Street 405 V7J 1H3 None https://rentals.ca/north-vancouver/1471-hunter-street-1 https://rentals.ca/north-vancouver/condos?active-listing-id=612435 6090785 proper 1471 Hunter Street highlighted 1 condo condo 2022-10-07T17:34:22.061Z [3400.0, 3400.0] [2.0, 2.0] [2.0, 2.0] [920.0, 920.0] 49.310450 -123.033497 -123.033497 49.310450 None https://images.rentals.ca/property-pictures/large/north-vancouver-bc/612435/condo-13068950.jpg 4032 4032 11 None https://images.rentals.ca/property-pictures/large/north-vancouver-bc/612435/condo-13068950.jpg 1440 1080 https://images.rentals.ca/property-pictures/medium/north-vancouver-bc/612435/condo-13068950.jpg 1200 900 https://images.rentals.ca/property-pictures/small/north-vancouver-bc/612435/condo-13068950.jpg 480 360
1 632195 None 5475 199A Street None V3A 1J7 None https://rentals.ca/langley/5475-199a-street https://rentals.ca/langley?active-listing-id=632195 6049430 proper 5475 199A Street highlighted 1 town-house town house 2022-10-05T18:55:29.314Z [2950.0, 2950.0] [3.0, 3.0] [2.5, 2.5] [1445.0, 1445.0] 49.101963 -122.670668 -122.670668 49.101963 None https://images.rentals.ca/property-pictures/large/langley-bc/632195/town-house-13671405.jpg 5452 5452 11 None https://images.rentals.ca/property-pictures/large/langley-bc/632195/town-house-13671405.jpg 1620 1080 https://images.rentals.ca/property-pictures/medium/langley-bc/632195/town-house-13671405.jpg 1350 900 https://images.rentals.ca/property-pictures/small/langley-bc/632195/town-house-13671405.jpg 540 360
2 629540 Hub Place 1649 E Broadway None None (604) 714-4794 https://rentals.ca/vancouver/1649-e-broadway https://rentals.ca/vancouver/apartments?active-listing-id=629540 6173331 proper 1649 E Broadway highlighted 1 apartment apartment 2022-10-05T09:08:12.408Z [1975.0, 4050.0] [0.0, 3.0] [1.0, 2.0] [362.0, 1009.0] 49.262622 -123.071263 -123.071263 49.262622 None https://images.rentals.ca/property-pictures/large/vancouver-bc/629540/apartment-13568421.jpg 1280 1280 11 None https://images.rentals.ca/property-pictures/large/vancouver-bc/629540/apartment-13568421.jpg 1280 853 https://images.rentals.ca/property-pictures/medium/vancouver-bc/629540/apartment-13568421.jpg 1280 853 https://images.rentals.ca/property-pictures/small/vancouver-bc/629540/apartment-13568421.jpg 540 360
3 468498 Arbutus Residences 4188 Yew Street None None (604) 879-8407 https://rentals.ca/vancouver/4188-yew-street https://rentals.ca/vancouver/apartments?active-listing-id=468498 6173331 proper 4188 Yew Street highlighted 1 apartment apartment 2022-08-23T09:00:21.332Z [2330.0, 10895.0] [0.5, 4.0] [1.0, 3.0] [455.0, 2220.0] 49.248762 -123.154540 -123.154540 49.248762 None https://images.rentals.ca/property-pictures/large/vancouver-bc/468498/apartment-5596387.jpg 1727 1727 11 None https://images.rentals.ca/property-pictures/large/vancouver-bc/468498/apartment-5596387.jpg 1619 1080 https://images.rentals.ca/property-pictures/medium/vancouver-bc/468498/apartment-5596387.jpg 1349 900 https://images.rentals.ca/property-pictures/small/vancouver-bc/468498/apartment-5596387.jpg 540 360
4 293930 9304 Salish Court 9304 Salish Court None None (778) 960-6568 https://rentals.ca/burnaby/9304-salish-court https://rentals.ca/burnaby/apartments?active-listing-id=293930 5911606 proper 9304 Salish Court paid-feed 1 apartment apartment 2022-10-10T09:11:13.322Z [2402.0, 2402.0] [1.0, 1.0] [1.0, 1.0] [784.0, 784.0] 49.251380 -122.901761 -122.901761 49.251380 None https://images.rentals.ca/property-pictures/large/burnaby-bc/293930/apartment-2685394.jpg 1727 1727 11 None https://images.rentals.ca/property-pictures/large/burnaby-bc/293930/apartment-2685394.jpg 1619 1080 https://images.rentals.ca/property-pictures/medium/burnaby-bc/293930/apartment-2685394.jpg 1349 900 https://images.rentals.ca/property-pictures/small/burnaby-bc/293930/apartment-2685394.jpg 540 360
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
195 636694 None 6628 120 Street 1 V3W 1T7 (604) 356-9352 https://rentals.ca/surrey/6628-120-street https://rentals.ca/surrey?active-listing-id=636694 6159905 proper 6628 120 Street None 1 private-room private room 2022-10-06T11:14:04.174Z [700.0, 700.0] [3.0, 3.0] [2.0, 2.0] [1000.0, 1000.0] 49.123305 -122.889203 -122.889203 49.123305 None https://images.rentals.ca/property-pictures/large/surrey-bc/636694/private-room-13739433.jpg 2448 2448 11 None https://images.rentals.ca/property-pictures/large/surrey-bc/636694/private-room-13739433.jpg 810 1080 https://images.rentals.ca/property-pictures/medium/surrey-bc/636694/private-room-13739433.jpg 675 900 https://images.rentals.ca/property-pictures/small/surrey-bc/636694/private-room-13739433.jpg 270 360
196 636691 None 138 East Hastings Street None V6A 1N6 (236) 237-3275 https://rentals.ca/vancouver/138-east-hastings-street https://rentals.ca/vancouver/apartments?active-listing-id=636691 6173331 proper 138 East Hastings Street None 1 apartment apartment 2022-10-06T11:13:44.397Z [2100.0, 2100.0] [1.0, 1.0] [1.0, 1.0] [468.0, 468.0] 49.281213 -123.101304 -123.101304 49.281213 None https://images.rentals.ca/property-pictures/large/vancouver-bc/636691/apartment-13739376.jpg 639 639 11 None https://images.rentals.ca/property-pictures/large/vancouver-bc/636691/apartment-13739376.jpg 639 526 https://images.rentals.ca/property-pictures/medium/vancouver-bc/636691/apartment-13739376.jpg 639 526 https://images.rentals.ca/property-pictures/small/vancouver-bc/636691/apartment-13739376.jpg 437 360
197 636644 None 1526 Finlay Street 310 V4B 4L9 (604) 222-8425 https://rentals.ca/white-rock/1526-finlay-street-2 https://rentals.ca/white-rock/apartments?active-listing-id=636644 6180961 proper 1526 Finlay Street None 1 apartment apartment 2022-10-06T11:10:37.594Z [3500.0, 3500.0] [2.0, 2.0] [2.0, 2.0] [1070.0, 1070.0] 49.029799 -122.789784 -122.789784 49.029799 None https://images.rentals.ca/property-pictures/large/white-rock-bc/636644/apartment-13737096.jpg 2016 2016 11 None https://images.rentals.ca/property-pictures/large/white-rock-bc/636644/apartment-13737096.jpg 1440 1080 https://images.rentals.ca/property-pictures/medium/white-rock-bc/636644/apartment-13737096.jpg 1200 900 https://images.rentals.ca/property-pictures/small/white-rock-bc/636644/apartment-13737096.jpg 480 360
198 636647 None Highway 99 None None (604) 219-7378 https://rentals.ca/richmond/highway-99 https://rentals.ca/richmond/condos?active-listing-id=636647 6122085 proper Highway 99 None 1 condo condo 2022-10-06T11:07:45.240Z [3100.0, 3100.0] [2.0, 2.0] [1.0, 1.0] [720.0, 720.0] 49.170432 -123.086936 -123.086936 49.170432 None https://images.rentals.ca/property-pictures/large/richmond-bc/636647/condo-13737142.jpg 4032 4032 11 None https://images.rentals.ca/property-pictures/large/richmond-bc/636647/condo-13737142.jpg 1440 1080 https://images.rentals.ca/property-pictures/medium/richmond-bc/636647/condo-13737142.jpg 1200 900 https://images.rentals.ca/property-pictures/small/richmond-bc/636647/condo-13737142.jpg 480 360
199 594287 14858 60 Avenue 14858 60 Avenue None V3S 1R8 (604) 531-1111 https://rentals.ca/surrey/14858-60-avenue-8 https://rentals.ca/surrey/apartments?active-listing-id=594287 6159905 proper 14858 60 Avenue None 1 apartment apartment 2022-10-06T09:25:21.357Z [3530.0, 3530.0] [3.0, 3.0] [2.0, 2.0] [1217.0, 1217.0] 49.111630 -122.810380 -122.810380 49.111630 None https://images.rentals.ca/property-pictures/large/surrey-bc/594287/apartment-7406506.jpg 906 906 11 None https://images.rentals.ca/property-pictures/large/surrey-bc/594287/apartment-7406506.jpg 906 605 https://images.rentals.ca/property-pictures/medium/surrey-bc/594287/apartment-7406506.jpg 906 605 https://images.rentals.ca/property-pictures/small/surrey-bc/594287/apartment-7406506.jpg 539 360
Documentation for Requests: https://requests.readthedocs.io/en/latest/
And for pandas: https://pandas.pydata.org/docs/