Trying to catch a weibo data recently, but found that using selenium, requests the request will appear after a period of time the problem of a status code of 418, oneself also added a cookie pool, because there is no change proxy IP issues? Watching a couple of git star more weibo crawler project as if they would not have this problem, and online also didn't find relevant explanation, now temporarily can only each request wait for 2 to 3 seconds, the speed is too slow a day..

CodePudding user response:

And I can't wait for 3 seconds, also in 418

CodePudding user response:

Gripper last year, I also met my crawl weibo data was quite many have not seen the problem now just need to climb some new data validation results found up to a little hurry up there will be 418 and not available in the Internet may weibo new strengthened the climb? I think again way can you solve it

CodePudding user response:

reference 4 floor weixin_43945615 response:

gripper last year, I also met my crawl weibo data was quite many have not seen this problem now need to crawl just found some new data validation results to a little hurry up there will be 418 and may not available in online weibo new strengthened the climb? I think again way can you solve it

Not yet, temporarily can only reduce the speed...

CodePudding user response:

Bosses, have you solved? The problem of 418

CodePudding user response:

Bosses solved? With the agent

CodePudding user response:

Today, is the question before I add the headers, each request headers using fake_useragent structure plus cookie pool, now there are no more questions, don't have 418 problems,

CodePudding user response:

As well as how to solve...

CodePudding user response:

Random header + IP pool