return _default_decoder.decode(s) error after reading json file in python-CodePudding

url = 'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment&cityName=Delhi-NCR'
result = requests.get(url)
data = result.json()

I'm trying to execute the above code but getting the following error. Is there any other module or library that i need to import or is it some syntax error? I've tried importing it by BeautifulSoup but since it's a dynamically loading page, it won't work.

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
File ~\anaconda3\lib\site-packages\requests\models.py:910, in Response.json(self, **kwargs)
    909 try:
--> 910     return complexjson.loads(self.text, **kwargs)
    911 except JSONDecodeError as e:
    912     # Catch JSON-related errors and raise as requests.JSONDecodeError
    913     # This aliases json.JSONDecodeError and simplejson.JSONDecodeError

File ~\anaconda3\lib\json\__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:

File ~\anaconda3\lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()

File ~\anaconda3\lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 2 column 1 (char 1)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
Input In [14], in <cell line: 3>()
      1 url = 'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment&cityName=Delhi-NCR'
      2 result = requests.get(url)
----> 3 data = result.json()

File ~\anaconda3\lib\site-packages\requests\models.py:917, in Response.json(self, **kwargs)
    915     raise RequestsJSONDecodeError(e.message)
    916 else:
--> 917     raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

CodePudding user response：

That's because the response content is an html. Do print(result.text) and you'll see :-D

Check if a different url will give you a json, otherwise, you may need to use BeautifulSoup or Scrapy to parse the html.

CodePudding user response：

That page is pulling its results from an API via an XHR call (check Dev Tools - Network tab). You need to scrape that particular endpoint, to get the json data. Here is one way to do it:

import requests

url = 'https://www.magicbricks.com/mbsrp/propertySearch.html?editSearch=Y&category=S&propertyType=10002,10003,10021,10022&city=3327-1001011&page=2&groupstart=30&offset=0&maxOffset=13&sortBy=premiumRecent&pType=10002,10003,10021,10022&isNRI=Y'

r = requests.get(url)
print(r.json())

This returns a (fairly big) JSON response:

{'nsrResultList': None, 'resultList': [{'encId': 'VNJDDhBvWUJzpSvf uAgZw==', 'possStatusD': 'Ready to Move', 'pmtSource': 'android', 'cntvl': '100', 'url': '2-BHK-940-Sq-ft-Multistorey-Apartment-FOR-Sale-K-R-Puram-in-Bangalore&id=4d423630333031323033', 'floorNo': '2', 'noBfCt': 0, 'st': 516, 'modstat': '20205', 'lCtDate': 1660792748000, 'appVer': 350, 'appSub': 'Y', 'deviceType': 'and', 'sendbirdId': '487111c6-5cb4-49e6-a35a-2dfa97bd6b62dipa6275', 'lastAccessDate': 1655404200000, 'projectSocietyLink': 'teja-nivas-k-r-puram-bangalore-pdpid-4d4235313830303531', 'sipro': '52063429 61161925 62018289 59395151', 'comFlag': 'N', 'appExc': 'N', 'cScore': '100', 'postDateT': '2022-08-11T23:54:46.000Z', 'endDateT': '2022-12-09T23:54:46.000Z', 'pl': 'Y', 'luxAmenities': '12223 12216', 'luxAmenitiesD': 'RO Water System,Visitor Parking', 'appovedAuthC': 'ABADev', 'nriPref': 'Y', 'enc_Id': 'VNJDDhBvWUKzvQzSxKfkaA==', 'cowDays': 0, 'price': 4200000, 'priceD': '42 Lac',....}