I have some code parsing a specific part of a dynamic webpage. Earlier I was doing it with scrapingant and it was working fine but due to the credits system it uses I decided to move to selenium. But when I try to run my code with selenium, instead of getting the dynamically generated HTML content, it just gives me the useless HTML that non-browser requests get, so it's not finding any of the elements that my code was finding before using srapingant. Any ideas? My code is below
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url='https://www.mesitis.com.cy/Search.aspx?&isAsc=0&isRent=0&districts=Lefkosia&status=1&apartFloor=&types=0dfe6c47-04c5-e511-ae61-a4badb3ceace&refno=&priceFrom=170000&priceTo=260000&priceRentFrom=0&priceRentTo=10000&areaFrom=0&areaTo=1000&intAreaFrom=130&intAreaTo=500&densityFrom=0&densityTo=200¤tPage=1'
# instance of Options class allows us to configure Headless Chrome
options = Options()
# this parameter tells Chrome that it should be run without UI (Headless)
options.headless = True
# initializing webdriver for Chrome with our options
driver = webdriver.Chrome(options=options, executable_path='C:\Downloads\chromedriver_win32\chromedriver.exe')
# getting webpage - this is where something seems to be going wrong, and I don't get the dynamically generated HTML content
driver.get(url)
page_source=driver.page_source
soup = BeautifulSoup(page_source,"html.parser")
body=soup.find('body')
properties=body.find('div', attrs={'class':'row properties-list'})
#find links of each property
allinks=properties.find_all('h3')
for d in allinks:
print('https://www.mesitis.com.cy/' d.a.get('href'))
CodePudding user response：
Here is one way of obtaining that data, without the overheads of Selenium: it's doable using Requests only:
import requests
import pandas as pd
import json
url = 'https://www.mesitis.com.cy/Search.aspx/SearchPropertiesFast'
big_df = pd.DataFrame()
headers = {
'origin': 'https://www.mesitis.com.cy',
'content-type': 'application/json; charset=UTF-8',
'sec-fetch-site': 'same-origin',
'x-requested-with': 'XMLHttpRequest',
'accept-language': 'en-US,en;q=0.9',
'accept': 'application/json, text/javascript, */*; q=0.01',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36',
}
payload = '{"formVars":[{"name":"__VIEWSTATE","value":"qDd5lhcP4H3kN2jINvhRjt92XZj6ix9IaKWrw9oIHKjw6zeDrrw2zWsQbGcDIZB/6U buWRwKUTyFye1K8vyTDeTj8MbT xSputXV4722aOhy672wwqzKKKDppNXgYLEZoM1aO0tlGNuGZrJ1EKT30FF2345a6oUPfKdrzHPyWAzxoceJ2/qFVuGQXMjAc/i"},{"name":"__VIEWSTATEGENERATOR","value":"BBBC20B8"},{"name":"__EVENTVALIDATION","value":"/8kjpsksY0pg1DuYQRFVE06InIT4LHDuZT8OFTY/pxYB3czd0Ukmi9thbd9XYDPNRXLkdm oJzu0U6g3bqVMZU5Azgj6C2V/LyMKsLq4tYuQ KE/GKWokA74mPf53qg 8tJ75T92dXShsE32JMlhiQ=="},{"name":"isAsc","value":"0"},{"name":"isRent","value":"0"},{"name":"districts","value":"Lefkosia"},{"name":"status","value":"1"},{"name":"apartFloor","value":""},{"name":"types","value":"0dfe6c47-04c5-e511-ae61-a4badb3ceace"},{"name":"refno","value":""},{"name":"priceFrom","value":170000},{"name":"priceTo","value":260000},{"name":"priceRentFrom","value":0},{"name":"priceRentTo","value":10000},{"name":"areaFrom","value":0},{"name":"areaTo","value":1000},{"name":"intAreaFrom","value":130},{"name":"intAreaTo","value":500},{"name":"densityFrom","value":0},{"name":"densityTo","value":200},{"name":"currentPage","value":1}]}'
r = requests.post(url, data = payload, headers=headers)
data_obj = json.loads(r.json()['d'])
df = pd.json_normalize(data_obj, record_path=['DataObject'])
df.to_csv('houses.csv')
print(df)
Result printed in terminal:
Price id ReferenceNo Title Area Type Sold IsRent PlotArea ProjectID ... LastUpdated propImageID IsProject District CoveredArea VAT IsNew PropertyId ApartFloor EnergyEfficiency
0 170000.0 cf241874-a168-ec11-aa4c-246e960cbc4d 16365 2 Bedroom Old House For Sale In Kaimakli Kaimakli House/Villa 0 False 270 None ... 2022-06-17T06:55:26.247 00000000-0000-0000-0000-000000000000 0 Lefkosia 0 0 ce241874-a168-ec11-aa4c-246e960cbc4d None 20
1 190000.0 cd1a0b90-07aa-ec11-aa4c-246e960cbc4d 18001 3 Bedroom House For Sale In Anthoupoli, Nicosia Anthoupoli House/Villa 1 False 0 None ... 2022-10-13T10:36:39.8 09667ce7-07aa-ec11-aa4c-246e960cbc4d 0 Lefkosia 141 1 288943c7-d6a9-ec11-aa4c-246e960cbc4d None 80
2 200000.0 c42dc379-5f67-e811-bb4e-a4badb3ceace 2605 2 Storey Modern 3 Bedroom House For Sale In Da... Dali House/Villa 0 False 210 None ... 2021-06-20T15:09:05.183 f37cf839-6667-e811-bb4e-a4badb3ceace 0 Lefkosia 180 0 c32dc379-5f67-e811-bb4e-a4badb3ceace None 91
3 200000.0 6370763d-61ab-e811-b319-a4badb3ceacd 2962 Very Nice And Spacious 3 Bedroom Detached Hous... Lakatameia House/Villa 1 False 0 None ... 2022-10-13T10:36:25.23 8384a75b-61ab-e811-b319-a4badb3ceacd 0 Lefkosia 170 1 6270763d-61ab-e811-b319-a4badb3ceacd None 91
4 200000.0 7da50193-266b-e811-bb4e-a4badb3ceace 2633 3 Bedroom Under Construction Detached 4 Houses... Tseri House/Villa 0 False 173 None ... 2021-06-20T15:09:38.88 c1b54437-0d81-e811-ae53-a4badb3ceacd 0 Lefkosia 150 VAT 0 7ca50193-266b-e811-bb4e-a4badb3ceace None 100
5 210000.0 96d0fb0a-89bd-ec11-aa4e-246e960cbc4d 18927 3 Bedroom House For Sale In Agios Dometios, Ni... Agios Dometios House/Villa 0 False 293 None ... 2022-08-08T13:20:09.877 d1fadd0c-8ebd-ec11-aa4e-246e960cbc4d 0 Lefkosia 140 0 95d0fb0a-89bd-ec11-aa4e-246e960cbc4d None 38
6 211000.0 1881b78e-52d7-ec11-aa4e-246e960cbc4d 19481 4 Bedroom House For Sale In Archangelos, Nicosia Archangelos House/Villa 0 False 296 None ... 2022-08-10T09:32:52.27 e0c3a52d-53d7-ec11-aa4e-246e960cbc4d 0 Lefkosia 130 0 1781b78e-52d7-ec11-aa4e-246e960cbc4d None 68
7 215000.0 eaa2b630-e685-ec11-aa4c-246e960cbc4d 16834 In Excellent Location 3 Bedrooms House In Arch... Archangelos House/Villa 0 False 180 None ... 2022-10-10T13:53:48.94 95726746-e685-ec11-aa4c-246e960cbc4d 0 Lefkosia 160 1 e9a2b630-e685-ec11-aa4c-246e960cbc4d None 80
8 220000.0 2ad40da2-a190-ec11-aa4c-246e960cbc4d 17075 Incomplete residential development in Politiko... Politiko House/Villa 0 False 0 None ... 2022-02-18T10:00:43.46 2bd40da2-a190-ec11-aa4c-246e960cbc4d 0 Lefkosia 509 0 29d40da2-a190-ec11-aa4c-246e960cbc4d None 0
9 220000.0 a19ad42e-cd59-e911-8b16-a4badb3ceacd 3846 3 Bedroom House For Sale In Spilia With Great ... Spilia House/Villa 0 False 0 None ... 2022-05-25T11:57:46.77 9214ee5b-df59-e911-8b16-a4badb3ceacd 0 Lefkosia 0 0 a09ad42e-cd59-e911-8b16-a4badb3ceacd None 68
10 225000.0 4bf71317-9cee-e911-b59c-a4badb3ceacd 5397 3 Bedroom House For Sale In Strovolos, Nicosia Strovolos House/Villa 0 False 280 None ... 2022-05-16T06:40:32.417 491a8e48-a6ee-e911-b59c-a4badb3ceacd 0 Lefkosia 149 0 4af71317-9cee-e911-b59c-a4badb3ceacd None 20
11 225000.0 082e2088-7baf-e911-935a-a4badb3ceacd 4738 4 Bedroom House For Sale In Nisou, Nicosia Nisou House/Villa 0 False 210 None ... 2021-09-06T14:05:58.267 dfed3499-7baf-e911-935a-a4badb3ceacd 0 Lefkosia 225 0 072e2088-7baf-e911-935a-a4badb3ceacd None 91
12 230000.0 8a978c4b-7383-ec11-aa4c-246e960cbc4d 16788 3 Bedroom Upper For Sale In Strovolos, Nicosia Strovolos House/Villa 0 False 0 None ... 2022-02-03T14:20:16.57 a7df5018-f384-ec11-aa4c-246e960cbc4d 0 Lefkosia 190 0 89978c4b-7383-ec11-aa4c-246e960cbc4d None 68
13 230000.0 960d522b-dc39-e911-a389-a4badb3ceacd 3710 Very Nice 3 Bedroom House For Sale Very Close ... Kakopetria House/Villa 0 False 0 None ... 2022-03-09T14:51:47.3 00000000-0000-0000-0000-000000000000 0 Lefkosia 0 0 950d522b-dc39-e911-a389-a4badb3ceacd None 80
14 234000.0 1c3cbfed-2030-ec11-aa4c-246e960cbc4d 15488 5 Bedroom House For Sale In Spilia, Nicosia Spilia House/Villa 0 False 762 None ... 2022-05-25T11:57:23.79 dab17ebe-2130-ec11-aa4c-246e960cbc4d 0 Lefkosia 180 0 1b3cbfed-2030-ec11-aa4c-246e960cbc4d None 20
15 235000.0 b306f83b-7723-ed11-aa4e-246e960cbc4d 20722 4 Bedroom House For Sale In Lakatamia Lakatameia House/Villa 0 False 500 None ... 2022-10-06T12:52:50.527 d36a70ee-8623-ed11-aa4e-246e960cbc4d 0 Lefkosia 180 1 b206f83b-7723-ed11-aa4e-246e960cbc4d None 91
16 235000.0 9925f398-c76d-ea11-a1df-a4badb3ceacd 7596 3 Bedroom House For Sale In Tseri, Nicosia Tseri House/Villa 0 False 0 None ... 2021-08-03T14:02:02.54 636187c3-fe6f-ea11-a1df-a4badb3ceacd 0 Lefkosia 170 0 9825f398-c76d-ea11-a1df-a4badb3ceacd None 91
17 235000.0 b536908d-d0a7-e911-935a-a4badb3ceacd 4605 3 Bedroom House/Villa For Sale in Dali, Nicosia Dali House/Villa 0 False 293 None ... 2022-10-13T11:01:48.23 4c888960-d3a7-e911-935a-a4badb3ceacd 0 Lefkosia 180 1 b436908d-d0a7-e911-935a-a4badb3ceacd None 80
18 238000.0 e2519c9f-7188-e611-878e-a4badb3ceace 1450 3 Bedroom Detached House For Sale In Geri, Ni... Geri House/Villa 0 False 200 None ... 2022-02-18T09:06:00.217 c831a074-def9-eb11-aa49-246e960cbc4d 0 Lefkosia 165 0 e1519c9f-7188-e611-878e-a4badb3ceace None 91
19 238000.0 bfe95fce-0959-e811-bb4e-a4badb3ceace 2547 A Very Nice 3 Bedroom Beautiful House For Sale... Kalo Chorio Oreinis House/Villa 0 False 0 None ... 2021-06-28T08:59:28.997 48c399a4-9568-e811-bb4e-a4badb3ceace 0 Lefkosia 230 0 bee95fce-0959-e811-bb4e-a4badb3ceace None 68
20 238350.0 fcfdc748-7990-ea11-a1df-a4badb3ceacd 8196 3 Bedroom House For Sale In Latsia, Nicosia Latsia (Lakkia) House/Villa 0 False 256 None ... 2022-06-23T09:23:39.51 f18a96be-d5f2-ec11-aa4e-246e960cbc4d 0 Lefkosia 143 0 fbfdc748-7990-ea11-a1df-a4badb3ceacd None 80
21 239000.0 3f5369dc-8bf0-ec11-aa4e-246e960cbc4d 20038 4 Bedroom House For Sale In Dali, Nicosia Dali House/Villa 0 False 258 None ... 2022-06-30T11:56:33.08 00000000-0000-0000-0000-000000000000 0 Lefkosia 190 0 3e5369dc-8bf0-ec11-aa4e-246e960cbc4d None 80
22 240000.0 93eac30c-cd29-ed11-aa4e-246e960cbc4d 20805 3 Bedroom House For Sale In Lakatameia, Anthou... Lakatameia House/Villa 0 False 159 None ... 2022-09-07T07:56:28.5 6e6adaea-692e-ed11-aa4e-246e960cbc4d 0 Lefkosia 143 1 92eac30c-cd29-ed11-aa4e-246e960cbc4d None 80
23 243500.0 753a3401-d75b-ea11-a1df-a4badb3ceacd 7339 3 Bedroom House For Sale Ιn Tseri, Nicosia Tseri House/Villa 0 False 700 None ... 2022-06-09T12:06:17.957 001fea44-d75b-ea11-a1df-a4badb3ceacd 0 Lefkosia 227 VAT 0 743a3401-d75b-ea11-a1df-a4badb3ceacd None 91
24 245000.0 0fd2cd26-9e23-ed11-aa4e-246e960cbc4d 20733 3 Bedroom Houses For Sale In Pera Chorio, Nicosia Pera Chorio House/Villa 0 False 581 None ... 2022-10-06T12:42:59.047 c1ad0059-9e23-ed11-aa4e-246e960cbc4d 0 Lefkosia 200 1 0ed2cd26-9e23-ed11-aa4e-246e960cbc4d None 68
25 245000.0 bb9bd1f8-c729-ed11-aa4e-246e960cbc4d 20804 3 Bedroom Plus Office House For Sale In Tseri,... Tseri House/Villa 0 False 215 None ... 2022-09-14T09:42:17.427 dac9ee2a-532f-ed11-aa4e-246e960cbc4d 0 Lefkosia 178 1 ba9bd1f8-c729-ed11-aa4e-246e960cbc4d None 91
26 245000.0 0de63389-f8ff-eb11-aa4a-246e960cbc4d 14744 3 Bedroom House For Sale In Kaimakli Kaimakli House/Villa 0 False 0 None ... 2021-08-24T07:46:06.86 2d97036a-faff-eb11-aa4a-246e960cbc4d 0 Lefkosia 180 0 0ce63389-f8ff-eb11-aa4a-246e960cbc4d None 38
27 250000.0 c6caa20d-b910-ec11-aa4a-246e960cbc4d 14865 3 Bedroom House For Sale In Mammari, Nicosia Mammari House/Villa 0 False 586 None ... 2022-08-02T07:51:38.443 1ebcd136-7311-ed11-aa4e-246e960cbc4d 0 Lefkosia 190 0 c5caa20d-b910-ec11-aa4a-246e960cbc4d None 80
28 250000.0 c1b1fcc5-9251-ea11-a1df-a4badb3ceacd 7090 Houses For Sale In Strovolos Nicosia Strovolos House/Villa 0 False 301 None ... 2022-10-06T12:17:01.56 22c7750a-9351-ea11-a1df-a4badb3ceacd 0 Lefkosia 322 1 c0b1fcc5-9251-ea11-a1df-a4badb3ceacd None 68
29 250000.0 b8823644-babb-eb11-aa46-246e960cbc4d 13710 4 Bedroom House For Sale In Dali, Nicosia Dali House/Villa 0 False 280 None ... 2022-07-04T12:39:25.063 66596a80-bbbb-eb11-aa46-246e960cbc4d 0 Lefkosia 190 0 b7823644-babb-eb11-aa46-246e960cbc4d None 100
30 250000.0 8cee923f-bc5c-ec11-aa4c-246e960cbc4d 16227 3 Bedroom House For Sale In Nisou Nisou House/Villa 0 False 456 None ... 2021-12-21T08:42:29.78 418ba317-cf61-ec11-aa4c-246e960cbc4d 0 Lefkosia 170 0 ff0e8d10-bc5c-ec11-aa4c-246e960cbc4d None 54
31 255000.0 ed27019b-f2db-ea11-a1df-a4badb3ceacd 8867 3 Bedroom House For Sale In Ag.Dometios, Nicosia Agios Dometios House/Villa 0 False 341 None ... 2021-09-08T13:05:39.39 e756e5d2-00dc-ea11-a1df-a4badb3ceacd 0 Lefkosia 143 0 ec27019b-f2db-ea11-a1df-a4badb3ceacd None 80
32 258000.0 c453e74a-8bba-e811-91b1-a4badb3ceacd 3048 4 Bedroom Under Construction Detached House Fo... Pera Chorio House/Villa 0 False 332 None ... 2021-06-19T12:41:37.69 fa1f8fa2-8bba-e811-91b1-a4badb3ceacd 0 Lefkosia 185 VAT 0 c353e74a-8bba-e811-91b1-a4badb3ceacd None 100
33 260000.0 58fcb64e-443a-ed11-aa4e-246e960cbc4d 21097 In Excellent Condition 3 Bedroom Plus Attic Ho... Anthoupoli House/Villa 0 False 190 None ... 2022-09-22T10:19:04.283 4e45580e-453a-ed11-aa4e-246e960cbc4d 0 Lefkosia 180 1 b9497873-8039-ed11-aa4e-246e960cbc4d None 91
34 260000.0 aef929d7-f7a3-eb11-aa45-246e960cbc4e 13205 5 Bedroom House For Sale In Lakatamia, Nicosia Lakatameia House/Villa 0 False 260 None ... 2022-06-09T13:03:20.523 271593f7-f7a3-eb11-aa45-246e960cbc4e 0 Lefkosia 293 0 adf929d7-f7a3-eb11-aa45-246e960cbc4e None 68
35 260000.0 d2106e0e-092a-ed11-aa4e-246e960cbc4d 20822 3 Bedroom House For Sale In Palouriotissa, Nic... Palouriotissa House/Villa 0 False 281 None ... 2022-09-06T14:36:45.15 bfd8748a-cd2a-ed11-aa4e-246e960cbc4d 0 Lefkosia 170 1 9aff1698-0f29-ed11-aa4e-246e960cbc4d None 80
36 rows × 21 columns
See documentation for Requests here: https://requests.readthedocs.io/en/latest/
And for Pandas: https://pandas.pydata.org/docs/