Hello, everyone, I am a website editor, once wanted to use creeper crawled take 600 article related to a certain keywords, source can be a website/WeChat/since the media, public how to programming?
Wish to play with you, I hope you give me a direction, writing code or leave WeChat private chat, will have certain reward, right

CodePudding user response:

Another problem is made public, I hope to find a day to article 10 hot, also want to use the crawler! Don't know where laid hands on him!

CodePudding user response:

Webdriver requests or scrapy
To find a record of the url, endures a TXT file
Keywords in a TXT file
Process:
1. Read the address
2. Input keywords
Access address
articles 3.4. Through the article address
5. Download the document to the specified location
Choose whether to open multiple threads or processes according to computer performance
According to the work you need to set up regular running
over