I'm beginner in Web Scrapping. I'm trying to get latitude and longitude from this web:

https://urbania.pe/inmueble/proyecto/ememhvin-proyecto-mariscal-castilla-lima-santiago-de-surco-tale-inmobiliaria-65659522

A part containing such data is:

<script type="text/javascript"> ==$0
    const POSTING = {{[...] 
    "locationId":"V1-B-4368","name":"Lima","label":"PROVINCIA","depth":1,"parent":{"locationId":"V1-A-111","name":"Peru urbania","label":"PAIS","depth":0,"parent":null,"acronym":null},"acronym":null},"acronym":null},"acronym":null},"postingGeolocation":{"geolocation":{"latitude":-12.133920500000000,"longitude":-77.014942900000000},
    [...]

<script>

I'm trying to do, but not works:

import requests
import pandas as pd
import re
from bs4 import BeautifulSoup
import time
from selenium.webdriver.common.keys import Keys
from selenium import webdriver
import urllib.parse

sa_key = 'ea69223fa47f72fac0907759' # TOKEN from a web 
sa_api = 'https://api.scrapingant.com/v2/general'

page='https://urbania.pe/inmueble/proyecto/ememhvin-proyecto-mariscal-castilla-lima-santiago-de-surco-tale-inmobiliaria-65659522'

qParams = {'url':page , 'x-api-key': sa_key}  #OJO: aqui tener cuidado con /proyecto/ y /clasificado/  , estructura para 1°
reqUrl = f'{sa_api}?{urllib.parse.urlencode(qParams)}'  

r = requests.get(reqUrl)
soup = BeautifulSoup(r.content, 'html.parser')

list_geolocalization=[]

# trying to get latitude and lingitude
geolocalization=soup.find_all('script',{'type': 'text/javascript'})

for tag in geolocalization:
    list_geolocalization.append(tag.find('latitude'))
    df_geolocalization=pd.DataFrame(list_geolocalization,columns = ["geolocalization"])

#other
lat, long=re.findall(r'(?is)("latitude":|"longitude":)([0-9.] )',geolocalization)

Can someone help me please? Thanks in advance!

CodePudding user response：

In this situation, You can take advantages of Regular Expression as follows:

import requests
from bs4 import BeautifulSoup
import re
import json

headers = {'User-Agent': 'Mozilla/5.0'}
url = "https://urbania.pe/inmueble/proyecto/ememhvin-proyecto-mariscal-castilla-lima-santiago-de-surco-tale-inmobiliaria-65659522"

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
print(response)
r = re.search(r'const POSTING = {.*}',str(soup))
if r:
    j = json.loads(r.group(0).replace('const POSTING = ', ''))
    lat = j.get('postingLocation', {}).get('postingGeolocation', {}).get('geolocation', {}).get('latitude')
    print(lat)
    long = j.get('postingLocation', {}).get('postingGeolocation', {}).get('geolocation', {}).get('longitude')
    print(long)
else:
    print("No match found.")

Outout:

-12.1339205
-77.0149429