BeautifulSoup4 return none with correct class name-CodePudding

I'm new to python and currently working on my FYP. The problem is that, the output I'm getting is none even if I believe I'm using the right parent div. The data I try to scrape is the name, price, and address. Below is the HTML:

Here is the code I try that return the none value.

from bs4 import BeautifulSoup
import requests

url = "https://www.edgeprop.my/buy/kelantan/all-residential?page=1"
page = requests.get(url)

soup = BeautifulSoup(page.content, 'html.parser')
lists = soup.find_all('div', class_="listing-col")

for list in lists:
    title = list.find('h3', class_="listing-name")
    price = list.find('h5', class_="listing-price")
    location = list.find('p', class_="listing-address")
    info = [title, price, location]
    print(info)

And the output I'm getting is [None, None, None]

CodePudding user response：

You could use their api to scrape structured data in JSON format.

{'property': [{'type_s': 'sale', 'agentplan_s': 'premium', 'field_prop_asset_i': 1100, 'mid_i': 1584670, 'uid_i': 40723, 'agent_name_s_lower': 'AZMAN MAT YASIN', 'agent_contact_s_lower': '0169213895', 'agent_profile_pic_s': 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC9Gcm9tYWtvazM0SUlJLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MTI1LCJoZWlnaHQiOjEyNSwiZml0IjoiY292ZXIifX19', 'agent_agency_logo_s': 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC8wNzBfMTYxODk4NDA4N19sb2dvX2xvZ29yaXpxcmVhbHR5cG5nLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJoZWlnaHQiOjQwLCJmaXQiOiJvdXRzaWRlIn19fQ==', 'status_i': 1, 'title_t': 'TERES SUPERLINK 3 TINGKAT TAMAN BAHAGIA KOTA BHARU', 'headline_t': 'TERES SUPERLINK 3 TINGKAT TAMAN BAHAGIA KOTA BHARU', 'created_i': 1644425292, 'updated_dt': '2022-04-07T00:48:01Z', 'changed_i': 1649263681, 'state_s_lower': 'Kelantan', 'district_s_lower': 'Kota Bharu', 'field_prop_area_unit_s_lower': 'sqft', 'field_prop_asking_price_d': 580000, 'field_prop_bathrooms_i': 7, 'field_prop_bedrooms_i': 7, 'field_prop_built_up_d': 3260, 'field_prop_built_up_price_pu_d': 177.91, 'field_prop_built_up_unit_s_lower': 'sqft', 'field_prop_car_parking_i': 2, 'field_prop_images_txt': ['https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MTU3XzVFZGdQcm81X3Bob3RvMTY0NDM5NzUxNzdfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MTgwXzVFZGdQcm81X3Bob3RvMTY0NDM5NzUxNzNfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MjA0XzVFZGdQcm81XzI2ODc4ODY0NF80ODQ1NTIzNDQyMTY1NTEyXzE1MDI1MDk0OTgwODcwOTUyMzRfbl8wLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MzAwLCJoZWlnaHQiOjIwMCwiZml0IjoiY292ZXIifX19', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MjA0XzVFZGdQcm81XzI2ODgwMjI1M180ODQ1NTIzMzM1NDk4ODU2XzQ1NTI4NzQ2MzIxODc3MTc5Nzlfbl8wLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MzAwLCJoZWlnaHQiOjIwMCwiZml0IjoiY292ZXIifX19', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MjA0XzVFZGdQcm81XzI2ODg1ODEzOF80ODQ1NTIzNDcyMTY1NTA5XzgxOTY3MjQ0OTQ2NTUyOTM2NzJfbl8wLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MzAwLCJoZWlnaHQiOjIwMCwiZml0IjoiY292ZXIifX19', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MjA1XzVFZGdQcm81XzI2OTAwNzM3Nl80ODQ1NTIzNjQ1NDk4ODI1XzEwMDQ5NTY0Mjc1Mzg1OTk3NTdfbl8wLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MzAwLCJoZWlnaHQiOjIwMCwiZml0IjoiY292ZXIifX19', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MjA1XzVFZGdQcm81XzI2OTM3ODk1OV80ODQ1NTIzNjY1NDk4ODIzXzg3MDczMTM1NjM3MjA2NTY1MTBfbl8wLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MzAwLCJoZWlnaHQiOjIwMCwiZml0IjoiY292ZXIifX19', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1MjYxXzVFZGdQcm81X3NpdGVwbGFuXzAuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjQ0NDI1Mjc4XzVFZGdQcm81X2Zsb29ycGxhbl8wLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MzAwLCJoZWlnaHQiOjIwMCwiZml0IjoiY292ZXIifX19'], 'field_prop_land_area_d': 1398, 'field_prop_land_area_sqft_d': 1398, 'field_prop_lease_term_s': 'fh', 'field_prop_postcode_i': 15300, 'field_prop_price_pu_d': 414.88, 'field_prop_street_t': 'Off Jalan Long Yunus', 'field_property_type_i': 36, 'field_property_subtype_i': 39, 'url_s': 'sale/1584670/kelantan/kota_bharu/landed/terracehouse/teres-superlink-3-tingkat-taman-bahagia-kota-bharu', 'score': 100.432274, 'shortlist': 0}, {'type_s': 'sale', 'agentplan_s': 'premium', 'field_prop_asset_i': 1100, 'mid_i': 1500124, 'uid_i': 40723, 'agent_name_s_lower': 'AZMAN MAT YASIN', 'agent_contact_s_lower': '0169213895', 'agent_profile_pic_s': 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC9Gcm9tYWtvazM0SUlJLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MTI1LCJoZWlnaHQiOjEyNSwiZml0IjoiY292ZXIifX19', 'agent_agency_logo_s': 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC8wNzBfMTYxODk4NDA4N19sb2dvX2xvZ29yaXpxcmVhbHR5cG5nLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJoZWlnaHQiOjQwLCJmaXQiOiJvdXRzaWRlIn19fQ==', 'status_i': 1, 'title_t': 'Rumah Teres 2 Tingkat Corner Lot', 'headline_t': 'Rumah Teres 2 Tingkat Corner Lot', 'created_i': 1633710812, 'updated_dt': '2022-04-07T00:33:01Z', 'changed_i': 1649262781, 'state_s_lower': 'Kelantan', 'district_s_lower': 'Kota Bharu', 'field_prop_area_unit_s_lower': 'sqft', 'field_prop_asking_price_d': 425000, 'field_prop_bathrooms_i': 3, 'field_prop_bedrooms_i': 4, 'field_prop_built_up_d': 1902, 'field_prop_built_up_price_pu_d': 223.45, 'field_prop_built_up_unit_s_lower': 'sqft', 'field_prop_car_parking_i': 2, 'field_prop_images_txt': ['https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzIxXzVFZGdQcm81X3Bob3RvMTYzMjY5NzU4NzdfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3Bob3RvMTYyMzc3MjE1NzFfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3Bob3RvMTYyMzc3MjE1ODJfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3Bob3RvMTYyMzc3MjE1OTFfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3Bob3RvMTYyMzc3MjE1OTJfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3Bob3RvMTYyMzc3MjE1OTNfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0=', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3Bob3RvMTYzMjY5NzU4NzEwXzAuanBlZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6MzAwLCJoZWlnaHQiOjIwMCwiZml0IjoiY292ZXIifX19', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3Bob3RvMTYzMjY5NzU4N18wLmpwZWciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjMwMCwiaGVpZ2h0IjoyMDAsImZpdCI6ImNvdmVyIn19fQ==', 'https://d5dkjitd3iude.cloudfront.net/eyJidWNrZXQiOiJ3d3cudGhlZWRnZXByb3BlcnR5LmNvbS5teSIsImtleSI6InMzZnMtcHVibGljXC80MDcyM18xNjMzNzEwNzk5XzVFZGdQcm81X3doYXRzYXBwaW1hZ2UyMDIxMDYxMWF0MTUuMzEuNTJfMC5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjozMDAsImhlaWdodCI6MjAwLCJmaXQiOiJjb3ZlciJ9fX0='], 'field_prop_land_area_d': 2497, 'field_prop_land_area_sqft_d': 2497, 'field_prop_lease_term_s': 'fh', 'field_prop_postcode_i': 15000, 'field_prop_price_pu_d': 170.2, 'field_prop_street_t': 'Taman Bahari', 'field_property_type_i': 36, 'field_property_subtype_i': 39, 'url_s': 'sale/1500124/kelantan/kota_bharu/landed/terracehouse/rumah-teres-2-tingkat-corner-lot', 'score': 100.42946, 'shortlist': 0},...]

Simply adjust parameter values &start=1&size=20 to your needs.

Example

import requests

url = 'https://www.edgeprop.my/jwdsonic/api/v1/property/search?&listing_type=sale&state=Kelantan&property_type=rl&start=1&size=20'
response = requests.get(url).json()

data = []

for item in response['property']:
    data.append([
        item['field_prop_asking_price_d'], 
        item['title_t'],
        item.get('field_prop_street_t')
    ])
data

Output

[[580000,
  'TERES SUPERLINK 3 TINGKAT TAMAN BAHAGIA KOTA BHARU',
  'Off Jalan Long Yunus'],
 [425000, 'Rumah Teres 2 Tingkat Corner Lot', 'Taman Bahari'],
 [195000,
  'PERUMAHAN TERES KG ARIL PAUH LIMA KOTA BHARU',
  'Jalan Kubang Tin - Pauh Lima'],
 [340000, '5576sqft Banglo Kampung Mengkuang Bachok Kelantan', None],
 [369000, 'TAMAN KAMUNTING, TANAH MERAH', 'TAMAN KAMUNTING'],
 [165000, 'Rumah Teres Taman Sri Warisan Pasir Puteh Kelantan', None],
 [1100000, 'Bandar Kota Bharu', 'Bandar Kota Bharu 15000 Kota Bharu Kelantan'],
 [1000000, 'Bungalow untuk Dijual di Kota Bharu Kelantan', 'kota bharu'],
 [3000000, 'Bandar Baru Pasir Mas', None],
 [295000, 'Rumah Teres 2 tingkat di Pasir Mas Untuk Dijual ', None],
 [175000,
  'Tanah Di Seberang Pasir Mas Kelantan Untuk Dijual',
  'jalan seberang pasir mas'],
...