Home > other >  Web page elements are dynamic loading, how to quickly find the corresponding dynamic link?
Web page elements are dynamic loading, how to quickly find the corresponding dynamic link?

Time:09-30

Himself is the crawler little sprout new, there are a lot of element dynamic crawl links don't know where I can find,
Such as the following url:
Url=http://sa.sogou.com/new-weball/page/sgs/epidemic
I want to climb in various provinces and cities and corresponding to the number of diagnose and cure
Don't know if it is a dynamic data at the beginning, with requests directly. The get method to crawl div tags only & lt; Div id="async" & gt;
,,
Just know this should be a dynamic loading data, but I can't find data source and no matter how to link where,,,
The console, and Network - XHR, JS, looked for it all, didn't find

# using selenium is simply crawl to ~ but I feel overqualified,,,
The from the selenium import webdriver
The import time

Driver=webdriver. Firefox (executable_path=r 'F: \ Application \ geckodriver exe')
Url="http://sa.sogou.com/new-weball/page/sgs/epidemic? '
Driver. The get (url)
Print (' waiting for loading 4 s... ')
Time. Sleep (4)
Data_list=driver. Find_elements_by_css_selector (' div. Data - box ')
Province_confirm_list=[]
For the data in data_list:
Province=data. Find_elements_by_tag_name (' span) [0]. Text
Cure_qty=data. Find_elements_by_tag_name (' span) [1]. The text
Province_confirm_list. Append ([province, cure_qty])

Also please let us know your younger brother great spirit:
How to distinguish the data is dynamic load?
If it is a dynamic loading, and how to find the corresponding dynamic request url

CodePudding user response:

Right click to view web pages source code

CodePudding user response:

reference 1/f, ice wind of reply:
right click to view page source code

Right, directly in the source code to find, if can't find the said as a dynamically loaded,
Find the corresponding url, you can use the developer tools, in the network to find the good in the XHR

CodePudding user response:

refer to the second floor Hajo_ response:
Quote: refer to 1st floor ice all over the sky the wind response:

Right click to view page source code

Right, directly in the source code to find, if can't find the said as a dynamically loaded,
Find the corresponding url, you can use the developer tools, in the network to find the good in the XHR

The source is looking into the data, but data encapsulation in Windows - initial.. A dictionary, now also don't know how to get out??
Can't find the information in the XHR
  • Related