Question such as, who help to write a crawler, it is difficult to? I am a layman, self-study crawler only learned a fur, rose less than baidu knows to posts,
Thank master first

CodePudding user response:

Code is as follows: first requests the library
The import requests
Keyword="lean manufacturing"
Try:
Kv={' wd ':' keyword '}
R=requests. Get (" http://www.baidu.com/s ", params=kv)
Print (r.r equest. Url)
R.r aise_for_status ()
Print (len (r.t ext))
Except:
Print (" crawl failure ")

Above lean production is crawl baidu keyword, baidu know the basic by the same token, the url will change back

CodePudding user response:

Tried it on, no,
The output is as follows:

 https://wappass.baidu.com/static/captcha/tuxing.html? & Ak=c27bbc89afca0463650ac9bde68ebe06 & amp; Backurl=https%3A%2F%2Fwww.baidu.com % 2 fs % 3 FWD % 3 dkeyword & amp; Logid=11094304378531508214 & amp; Signature=968 c037e092dda5b1c1b4fe6554c81bc & amp; Timestamp=1575735550 
1519 

The Process finished with exit code 0 

Open the link, to verify a picture, after manual verification, open like this:


I am trying to search the post preserved, warrior, please help, thank you very much,

CodePudding user response:

Somebody else wrote an example, Keyword is your "lean manufacturing", this is not so simple as you think, give you a code, you can use the
First, what do you want? Did you use the baidu search keywords, you want the links, or page content in detail, and finally to save what, the in the mind have a look,
Second, what operation? Request connection, whether there will be a dynamic loading, whether to need to turn pages, whether you need to enter the detail page, detail page format is unified, etc.,
Third, what techniques are used? With simple requests library can meet the requirements, how to turn the page, how to enter the detail page, how to locate the node collection content, save the data form, database, file, or object? Etc.
Better to outsource if they won't solve

CodePudding user response:

reference 3 floor water flows Dong spreading reply:

somebody else wrote an example, Keyword is your "lean manufacturing", this is not so simple as you think, give you a code, you can use the
First, what do you want? Did you use the baidu search keywords, you want the links, or page content in detail, and finally to save what, the in the mind have a look,
Second, what operation? Request connection, whether there will be a dynamic loading, whether to need to turn pages, whether you need to enter the detail page, detail page format is unified, etc.,
Third, what techniques are used? With simple requests library can meet the requirements, how to turn the page, how to enter the detail page, how to locate the node collection content, save the data form, database, file, or object? Etc.
If not solve, rather than outsourcing

Thank you,
I want to keep the search postings to a file,

Want to ask everybody to help, help me to write the the main parts, for example, the search of the "lean manufacturing" in the "baidu knows" the first page of the post, to help save to file,
If you don't too hard, just bother warrior help, really hard, and then another to find a way to
I'm not a professional to do this, is this kind of data is needed to do a report, thank you

CodePudding user response:

Or which blog is similar to the example in the core code, give directions, thank you master