Home > other > Web page elements are dynamic loading, how to quickly find the corresponding dynamic link?
Web page elements are dynamic loading, how to quickly find the corresponding dynamic link?
Time:09-30
Himself is the crawler little sprout new, there are a lot of element dynamic crawl links don't know where I can find, Such as the following url: Url=http://sa.sogou.com/new-weball/page/sgs/epidemic I want to climb in various provinces and cities and corresponding to the number of diagnose and cure Don't know if it is a dynamic data at the beginning, with requests directly. The get method to crawl div tags only & lt; Div id="async" & gt;
,, Just know this should be a dynamic loading data, but I can't find data source and no matter how to link where,,, The console, and Network - XHR, JS, looked for it all, didn't find
# using selenium is simply crawl to ~ but I feel overqualified,,, The from the selenium import webdriver The import time
Driver=webdriver. Firefox (executable_path=r 'F: \ Application \ geckodriver exe') Url="http://sa.sogou.com/new-weball/page/sgs/epidemic? ' Driver. The get (url) Print (' waiting for loading 4 s... ') Time. Sleep (4) Data_list=driver. Find_elements_by_css_selector (' div. Data - box ') Province_confirm_list=[] For the data in data_list: Province=data. Find_elements_by_tag_name (' span) [0]. Text Cure_qty=data. Find_elements_by_tag_name (' span) [1]. The text Province_confirm_list. Append ([province, cure_qty])
Also please let us know your younger brother great spirit: How to distinguish the data is dynamic load? If it is a dynamic loading, and how to find the corresponding dynamic request url