Python BeatifulSoup find all divs returning empty-CodePudding

I'm trying to get all house listings from a portuguese real estate agency website. I'm using the following small piece of code:

import requests
import bs4
import os

REAL_ESTATE_AGENCY_URL= os.getenv('REAL_ESTATE_AGENCY_URL')

response = requests.get(REAL_ESTATE_AGENCY_URL)
response.raise_for_status()
content = response.text

soup = bs4.BeautifulSoup(content, 'lxml')

all_listings = soup.find_all(name="div", class_="listing-card")
print(all_listings)

The same link I'm using is:

CodePudding user response：

You are returning empty result because the website is entirely depends on JavaScript.BeautifulSoup can't mimic data but You can grab data easily from api calls json response using only requests module.Here is the woriking example.

Script:

import requests
import json
body= {"filters":[{"field":"BusinessTypeID","value":"1","type":0},{"field":"NumberOfBedrooms","biggerThan":2,"type":4},{"field":"ListingTypeID","shouldValues":["11","1"],"type":2},{"field":"Region1ID","value":"78","type":0}],"sort":{"fieldToSort":"ContractDate","order":1}}
headers= {
    'content-type': 'application/json',
   }

api_url = "https://www.remax.pt/Api/Listing/MultiMatchSearch?page=1&searchValue=&size=20"


jsonData = requests.post(api_url, data=json.dumps(body), headers=headers).json()
#print(jsonData)
for item in jsonData['results']:
    price=item['listingPriceText']
    print(price)

Output: