Home > Software engineering >  Using headless selenium to scrape dynamic site - Why am I not getting the script-generated content?
Using headless selenium to scrape dynamic site - Why am I not getting the script-generated content?

Time:10-15

I have some code parsing a specific part of a dynamic webpage. Earlier I was doing it with scrapingant and it was working fine but due to the credits system it uses I decided to move to selenium. But when I try to run my code with selenium, instead of getting the dynamically generated HTML content, it just gives me the useless HTML that non-browser requests get, so it's not finding any of the elements that my code was finding before using srapingant. Any ideas? My code is below

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

url='https://www.mesitis.com.cy/Search.aspx?&isAsc=0&isRent=0&districts=Lefkosia&status=1&apartFloor=&types=0dfe6c47-04c5-e511-ae61-a4badb3ceace&refno=&priceFrom=170000&priceTo=260000&priceRentFrom=0&priceRentTo=10000&areaFrom=0&areaTo=1000&intAreaFrom=130&intAreaTo=500&densityFrom=0&densityTo=200&currentPage=1'

# instance of Options class allows us to configure Headless Chrome
options = Options()

# this parameter tells Chrome that it should be run without UI (Headless)
options.headless = True

# initializing webdriver for Chrome with our options
driver = webdriver.Chrome(options=options, executable_path='C:\Downloads\chromedriver_win32\chromedriver.exe')

# getting webpage - this is where something seems to be going wrong, and I don't get the dynamically generated HTML content
driver.get(url)
page_source=driver.page_source
soup = BeautifulSoup(page_source,"html.parser")

body=soup.find('body')

properties=body.find('div', attrs={'class':'row properties-list'})

#find links of each property
allinks=properties.find_all('h3')
for d in allinks:
    print('https://www.mesitis.com.cy/' d.a.get('href'))

CodePudding user response:

Here is one way of obtaining that data, without the overheads of Selenium: it's doable using Requests only:

import requests
import pandas as pd
import json

url = 'https://www.mesitis.com.cy/Search.aspx/SearchPropertiesFast'

big_df = pd.DataFrame()
headers = {
    'origin': 'https://www.mesitis.com.cy',
    'content-type': 'application/json; charset=UTF-8',
    'sec-fetch-site': 'same-origin',
    'x-requested-with': 'XMLHttpRequest',
    'accept-language': 'en-US,en;q=0.9',
    'accept': 'application/json, text/javascript, */*; q=0.01',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36',
}
payload = '{"formVars":[{"name":"__VIEWSTATE","value":"qDd5lhcP4H3kN2jINvhRjt92XZj6ix9IaKWrw9oIHKjw6zeDrrw2zWsQbGcDIZB/6U buWRwKUTyFye1K8vyTDeTj8MbT xSputXV4722aOhy672wwqzKKKDppNXgYLEZoM1aO0tlGNuGZrJ1EKT30FF2345a6oUPfKdrzHPyWAzxoceJ2/qFVuGQXMjAc/i"},{"name":"__VIEWSTATEGENERATOR","value":"BBBC20B8"},{"name":"__EVENTVALIDATION","value":"/8kjpsksY0pg1DuYQRFVE06InIT4LHDuZT8OFTY/pxYB3czd0Ukmi9thbd9XYDPNRXLkdm oJzu0U6g3bqVMZU5Azgj6C2V/LyMKsLq4tYuQ KE/GKWokA74mPf53qg 8tJ75T92dXShsE32JMlhiQ=="},{"name":"isAsc","value":"0"},{"name":"isRent","value":"0"},{"name":"districts","value":"Lefkosia"},{"name":"status","value":"1"},{"name":"apartFloor","value":""},{"name":"types","value":"0dfe6c47-04c5-e511-ae61-a4badb3ceace"},{"name":"refno","value":""},{"name":"priceFrom","value":170000},{"name":"priceTo","value":260000},{"name":"priceRentFrom","value":0},{"name":"priceRentTo","value":10000},{"name":"areaFrom","value":0},{"name":"areaTo","value":1000},{"name":"intAreaFrom","value":130},{"name":"intAreaTo","value":500},{"name":"densityFrom","value":0},{"name":"densityTo","value":200},{"name":"currentPage","value":1}]}'
r = requests.post(url, data = payload, headers=headers)  
data_obj = json.loads(r.json()['d'])
df = pd.json_normalize(data_obj, record_path=['DataObject'])
df.to_csv('houses.csv')
print(df)

Result printed in terminal:

    Price   id  ReferenceNo Title   Area    Type    Sold    IsRent  PlotArea    ProjectID   ... LastUpdated propImageID IsProject   District    CoveredArea VAT IsNew   PropertyId  ApartFloor  EnergyEfficiency
0   170000.0    cf241874-a168-ec11-aa4c-246e960cbc4d    16365   2 Bedroom Old House For Sale In Kaimakli    Kaimakli    House/Villa 0   False   270 None    ... 2022-06-17T06:55:26.247 00000000-0000-0000-0000-000000000000    0   Lefkosia    0       0   ce241874-a168-ec11-aa4c-246e960cbc4d    None    20
1   190000.0    cd1a0b90-07aa-ec11-aa4c-246e960cbc4d    18001   3 Bedroom House For Sale In Anthoupoli, Nicosia Anthoupoli  House/Villa 1   False   0   None    ... 2022-10-13T10:36:39.8   09667ce7-07aa-ec11-aa4c-246e960cbc4d    0   Lefkosia    141     1   288943c7-d6a9-ec11-aa4c-246e960cbc4d    None    80
2   200000.0    c42dc379-5f67-e811-bb4e-a4badb3ceace    2605    2 Storey Modern 3 Bedroom House For Sale In Da...   Dali    House/Villa 0   False   210 None    ... 2021-06-20T15:09:05.183 f37cf839-6667-e811-bb4e-a4badb3ceace    0   Lefkosia    180     0   c32dc379-5f67-e811-bb4e-a4badb3ceace    None    91
3   200000.0    6370763d-61ab-e811-b319-a4badb3ceacd    2962    Very Nice And Spacious 3 Bedroom Detached Hous...   Lakatameia  House/Villa 1   False   0   None    ... 2022-10-13T10:36:25.23  8384a75b-61ab-e811-b319-a4badb3ceacd    0   Lefkosia    170     1   6270763d-61ab-e811-b319-a4badb3ceacd    None    91
4   200000.0    7da50193-266b-e811-bb4e-a4badb3ceace    2633    3 Bedroom Under Construction Detached 4 Houses...   Tseri   House/Villa 0   False   173 None    ... 2021-06-20T15:09:38.88  c1b54437-0d81-e811-ae53-a4badb3ceacd    0   Lefkosia    150   VAT   0   7ca50193-266b-e811-bb4e-a4badb3ceace    None    100
5   210000.0    96d0fb0a-89bd-ec11-aa4e-246e960cbc4d    18927   3 Bedroom House For Sale In Agios Dometios, Ni...   Agios Dometios  House/Villa 0   False   293 None    ... 2022-08-08T13:20:09.877 d1fadd0c-8ebd-ec11-aa4e-246e960cbc4d    0   Lefkosia    140     0   95d0fb0a-89bd-ec11-aa4e-246e960cbc4d    None    38
6   211000.0    1881b78e-52d7-ec11-aa4e-246e960cbc4d    19481   4 Bedroom House For Sale In Archangelos, Nicosia    Archangelos House/Villa 0   False   296 None    ... 2022-08-10T09:32:52.27  e0c3a52d-53d7-ec11-aa4e-246e960cbc4d    0   Lefkosia    130     0   1781b78e-52d7-ec11-aa4e-246e960cbc4d    None    68
7   215000.0    eaa2b630-e685-ec11-aa4c-246e960cbc4d    16834   In Excellent Location 3 Bedrooms House In Arch...   Archangelos House/Villa 0   False   180 None    ... 2022-10-10T13:53:48.94  95726746-e685-ec11-aa4c-246e960cbc4d    0   Lefkosia    160     1   e9a2b630-e685-ec11-aa4c-246e960cbc4d    None    80
8   220000.0    2ad40da2-a190-ec11-aa4c-246e960cbc4d    17075   Incomplete residential development in Politiko...   Politiko    House/Villa 0   False   0   None    ... 2022-02-18T10:00:43.46  2bd40da2-a190-ec11-aa4c-246e960cbc4d    0   Lefkosia    509     0   29d40da2-a190-ec11-aa4c-246e960cbc4d    None    0
9   220000.0    a19ad42e-cd59-e911-8b16-a4badb3ceacd    3846    3 Bedroom House For Sale In Spilia With Great ...   Spilia  House/Villa 0   False   0   None    ... 2022-05-25T11:57:46.77  9214ee5b-df59-e911-8b16-a4badb3ceacd    0   Lefkosia    0       0   a09ad42e-cd59-e911-8b16-a4badb3ceacd    None    68
10  225000.0    4bf71317-9cee-e911-b59c-a4badb3ceacd    5397    3 Bedroom House For Sale In Strovolos, Nicosia  Strovolos   House/Villa 0   False   280 None    ... 2022-05-16T06:40:32.417 491a8e48-a6ee-e911-b59c-a4badb3ceacd    0   Lefkosia    149     0   4af71317-9cee-e911-b59c-a4badb3ceacd    None    20
11  225000.0    082e2088-7baf-e911-935a-a4badb3ceacd    4738    4 Bedroom House For Sale In Nisou, Nicosia  Nisou   House/Villa 0   False   210 None    ... 2021-09-06T14:05:58.267 dfed3499-7baf-e911-935a-a4badb3ceacd    0   Lefkosia    225     0   072e2088-7baf-e911-935a-a4badb3ceacd    None    91
12  230000.0    8a978c4b-7383-ec11-aa4c-246e960cbc4d    16788   3 Bedroom Upper For Sale In Strovolos, Nicosia  Strovolos   House/Villa 0   False   0   None    ... 2022-02-03T14:20:16.57  a7df5018-f384-ec11-aa4c-246e960cbc4d    0   Lefkosia    190     0   89978c4b-7383-ec11-aa4c-246e960cbc4d    None    68
13  230000.0    960d522b-dc39-e911-a389-a4badb3ceacd    3710    Very Nice 3 Bedroom House For Sale Very Close ...   Kakopetria  House/Villa 0   False   0   None    ... 2022-03-09T14:51:47.3   00000000-0000-0000-0000-000000000000    0   Lefkosia    0       0   950d522b-dc39-e911-a389-a4badb3ceacd    None    80
14  234000.0    1c3cbfed-2030-ec11-aa4c-246e960cbc4d    15488   5 Bedroom House For Sale In Spilia, Nicosia Spilia  House/Villa 0   False   762 None    ... 2022-05-25T11:57:23.79  dab17ebe-2130-ec11-aa4c-246e960cbc4d    0   Lefkosia    180     0   1b3cbfed-2030-ec11-aa4c-246e960cbc4d    None    20
15  235000.0    b306f83b-7723-ed11-aa4e-246e960cbc4d    20722   4 Bedroom House For Sale In Lakatamia   Lakatameia  House/Villa 0   False   500 None    ... 2022-10-06T12:52:50.527 d36a70ee-8623-ed11-aa4e-246e960cbc4d    0   Lefkosia    180     1   b206f83b-7723-ed11-aa4e-246e960cbc4d    None    91
16  235000.0    9925f398-c76d-ea11-a1df-a4badb3ceacd    7596    3 Bedroom House For Sale In Tseri, Nicosia  Tseri   House/Villa 0   False   0   None    ... 2021-08-03T14:02:02.54  636187c3-fe6f-ea11-a1df-a4badb3ceacd    0   Lefkosia    170     0   9825f398-c76d-ea11-a1df-a4badb3ceacd    None    91
17  235000.0    b536908d-d0a7-e911-935a-a4badb3ceacd    4605    3 Bedroom House/Villa For Sale in Dali, Nicosia Dali    House/Villa 0   False   293 None    ... 2022-10-13T11:01:48.23  4c888960-d3a7-e911-935a-a4badb3ceacd    0   Lefkosia    180     1   b436908d-d0a7-e911-935a-a4badb3ceacd    None    80
18  238000.0    e2519c9f-7188-e611-878e-a4badb3ceace    1450    3 Bedroom Detached House For Sale In Geri, Ni...    Geri    House/Villa 0   False   200 None    ... 2022-02-18T09:06:00.217 c831a074-def9-eb11-aa49-246e960cbc4d    0   Lefkosia    165     0   e1519c9f-7188-e611-878e-a4badb3ceace    None    91
19  238000.0    bfe95fce-0959-e811-bb4e-a4badb3ceace    2547    A Very Nice 3 Bedroom Beautiful House For Sale...   Kalo Chorio Oreinis House/Villa 0   False   0   None    ... 2021-06-28T08:59:28.997 48c399a4-9568-e811-bb4e-a4badb3ceace    0   Lefkosia    230     0   bee95fce-0959-e811-bb4e-a4badb3ceace    None    68
20  238350.0    fcfdc748-7990-ea11-a1df-a4badb3ceacd    8196    3 Bedroom House For Sale In Latsia, Nicosia Latsia (Lakkia) House/Villa 0   False   256 None    ... 2022-06-23T09:23:39.51  f18a96be-d5f2-ec11-aa4e-246e960cbc4d    0   Lefkosia    143     0   fbfdc748-7990-ea11-a1df-a4badb3ceacd    None    80
21  239000.0    3f5369dc-8bf0-ec11-aa4e-246e960cbc4d    20038   4 Bedroom House For Sale In Dali, Nicosia   Dali    House/Villa 0   False   258 None    ... 2022-06-30T11:56:33.08  00000000-0000-0000-0000-000000000000    0   Lefkosia    190     0   3e5369dc-8bf0-ec11-aa4e-246e960cbc4d    None    80
22  240000.0    93eac30c-cd29-ed11-aa4e-246e960cbc4d    20805   3 Bedroom House For Sale In Lakatameia, Anthou...   Lakatameia  House/Villa 0   False   159 None    ... 2022-09-07T07:56:28.5   6e6adaea-692e-ed11-aa4e-246e960cbc4d    0   Lefkosia    143     1   92eac30c-cd29-ed11-aa4e-246e960cbc4d    None    80
23  243500.0    753a3401-d75b-ea11-a1df-a4badb3ceacd    7339    3 Bedroom House For Sale Ιn Tseri, Nicosia  Tseri   House/Villa 0   False   700 None    ... 2022-06-09T12:06:17.957 001fea44-d75b-ea11-a1df-a4badb3ceacd    0   Lefkosia    227   VAT   0   743a3401-d75b-ea11-a1df-a4badb3ceacd    None    91
24  245000.0    0fd2cd26-9e23-ed11-aa4e-246e960cbc4d    20733   3 Bedroom Houses For Sale In Pera Chorio, Nicosia   Pera Chorio House/Villa 0   False   581 None    ... 2022-10-06T12:42:59.047 c1ad0059-9e23-ed11-aa4e-246e960cbc4d    0   Lefkosia    200     1   0ed2cd26-9e23-ed11-aa4e-246e960cbc4d    None    68
25  245000.0    bb9bd1f8-c729-ed11-aa4e-246e960cbc4d    20804   3 Bedroom Plus Office House For Sale In Tseri,...   Tseri   House/Villa 0   False   215 None    ... 2022-09-14T09:42:17.427 dac9ee2a-532f-ed11-aa4e-246e960cbc4d    0   Lefkosia    178     1   ba9bd1f8-c729-ed11-aa4e-246e960cbc4d    None    91
26  245000.0    0de63389-f8ff-eb11-aa4a-246e960cbc4d    14744   3 Bedroom House For Sale In Kaimakli    Kaimakli    House/Villa 0   False   0   None    ... 2021-08-24T07:46:06.86  2d97036a-faff-eb11-aa4a-246e960cbc4d    0   Lefkosia    180     0   0ce63389-f8ff-eb11-aa4a-246e960cbc4d    None    38
27  250000.0    c6caa20d-b910-ec11-aa4a-246e960cbc4d    14865   3 Bedroom House For Sale In Mammari, Nicosia    Mammari House/Villa 0   False   586 None    ... 2022-08-02T07:51:38.443 1ebcd136-7311-ed11-aa4e-246e960cbc4d    0   Lefkosia    190     0   c5caa20d-b910-ec11-aa4a-246e960cbc4d    None    80
28  250000.0    c1b1fcc5-9251-ea11-a1df-a4badb3ceacd    7090    Houses For Sale In Strovolos Nicosia    Strovolos   House/Villa 0   False   301 None    ... 2022-10-06T12:17:01.56  22c7750a-9351-ea11-a1df-a4badb3ceacd    0   Lefkosia    322     1   c0b1fcc5-9251-ea11-a1df-a4badb3ceacd    None    68
29  250000.0    b8823644-babb-eb11-aa46-246e960cbc4d    13710   4 Bedroom House For Sale In Dali, Nicosia   Dali    House/Villa 0   False   280 None    ... 2022-07-04T12:39:25.063 66596a80-bbbb-eb11-aa46-246e960cbc4d    0   Lefkosia    190     0   b7823644-babb-eb11-aa46-246e960cbc4d    None    100
30  250000.0    8cee923f-bc5c-ec11-aa4c-246e960cbc4d    16227   3 Bedroom House For Sale In Nisou   Nisou   House/Villa 0   False   456 None    ... 2021-12-21T08:42:29.78  418ba317-cf61-ec11-aa4c-246e960cbc4d    0   Lefkosia    170     0   ff0e8d10-bc5c-ec11-aa4c-246e960cbc4d    None    54
31  255000.0    ed27019b-f2db-ea11-a1df-a4badb3ceacd    8867    3 Bedroom House For Sale In Ag.Dometios, Nicosia    Agios Dometios  House/Villa 0   False   341 None    ... 2021-09-08T13:05:39.39  e756e5d2-00dc-ea11-a1df-a4badb3ceacd    0   Lefkosia    143     0   ec27019b-f2db-ea11-a1df-a4badb3ceacd    None    80
32  258000.0    c453e74a-8bba-e811-91b1-a4badb3ceacd    3048    4 Bedroom Under Construction Detached House Fo...   Pera Chorio House/Villa 0   False   332 None    ... 2021-06-19T12:41:37.69  fa1f8fa2-8bba-e811-91b1-a4badb3ceacd    0   Lefkosia    185   VAT   0   c353e74a-8bba-e811-91b1-a4badb3ceacd    None    100
33  260000.0    58fcb64e-443a-ed11-aa4e-246e960cbc4d    21097   In Excellent Condition 3 Bedroom Plus Attic Ho...   Anthoupoli  House/Villa 0   False   190 None    ... 2022-09-22T10:19:04.283 4e45580e-453a-ed11-aa4e-246e960cbc4d    0   Lefkosia    180     1   b9497873-8039-ed11-aa4e-246e960cbc4d    None    91
34  260000.0    aef929d7-f7a3-eb11-aa45-246e960cbc4e    13205   5 Bedroom House For Sale In Lakatamia, Nicosia  Lakatameia  House/Villa 0   False   260 None    ... 2022-06-09T13:03:20.523 271593f7-f7a3-eb11-aa45-246e960cbc4e    0   Lefkosia    293     0   adf929d7-f7a3-eb11-aa45-246e960cbc4e    None    68
35  260000.0    d2106e0e-092a-ed11-aa4e-246e960cbc4d    20822   3 Bedroom House For Sale In Palouriotissa, Nic...   Palouriotissa   House/Villa 0   False   281 None    ... 2022-09-06T14:36:45.15  bfd8748a-cd2a-ed11-aa4e-246e960cbc4d    0   Lefkosia    170     1   9aff1698-0f29-ed11-aa4e-246e960cbc4d    None    80
36 rows × 21 columns

​ ​See documentation for Requests here: https://requests.readthedocs.io/en/latest/

And for Pandas: https://pandas.pydata.org/docs/

  • Related