With requests. Get get page source code div tags didn't, so not to crawl-CodePudding

http://100wangke.com/
Input:
in the Middle EastHow should crawl?

CodePudding user response:

This will crawl data is web page dynamically loaded, so the information they need to crawl, look at the web page source code to know,

CodePudding user response:

Can click with selenium simulation

CodePudding user response:

refer to the second floor mathematical slasher response:

can use selenium simulate click

I use selenium analog input problem in click search, and then how to obtain?

CodePudding user response:

reference 1/f, try _except response:

this to crawl data is 1 page dynamically loaded, so the information they need to crawl, look at the web page source code to know,

after that I use selenium simulation search how to get?

CodePudding user response:

reference weixin_43770241 reply: 3/f

Quote: refer to the second floor mathematical slasher response:

Selenium can be used to simulate click

I use selenium analog input problem in click search, and then how to obtain?

You want to get what, just climb take corresponding part of the bai, what xpath parsing method, can be used

CodePudding user response:

reference 4 floor weixin_43770241 response:

Quote: reference 1/f, try _except response:

This will crawl data is web page dynamically loaded, so the information they need to crawl, look at the web page source code to know,

after that I use selenium simulation search how to get?

Sorry, I for this also not

CodePudding user response:

reference 5 floor mathematical slasher reply:

Quote: refer to the third floor weixin_43770241 response:

Quote: refer to the second floor mathematical slasher response:

Selenium can be used to simulate click

I use selenium analog input problem in click search, and then how to obtain?

You want to get what, just climb take corresponding part of the bai, what xpath parsing method, can use ah

convenient specific tell me what kind of code?

CodePudding user response:

refer to 7th floor weixin_43770241 response:

Quote: refer to fifth floor mathematical slasher response:

Quote: refer to the third floor weixin_43770241 response:

Quote: refer to the second floor mathematical slasher response:

Selenium can be used to simulate click

I use selenium analog input problem in click search, and then how to obtain?

You want to get what, just climb take corresponding part of the bai, what xpath parsing method, can use ah

convenient specific tell me what kind of code?

Two methods:
In selenium: keyword, click simulation, and then let the driver to sleep 1 second, check the answer of elements can be
2. Caught, find it request API, I looked at it a little, he sends an ajax request, which carries the two parameters "question" and "token", the question is do you want to enter the query keywords, token is he constructed a cipher text, you can be caught to Sources, will find a token of generating function, called ccall, compile and execute JavaScript code, in python generated token can

Teach them to fish, as teach them to fish