This IP, like the tide, have been made in bulk, the first time all in through bidding links, but later analysis and is not, the first is the consumer without exception, the second is the IP segment seems to be baidu inc., but there is a phenomenon of rejected this idea, I because general spider crawling site, in 51 la statistics are statistics don't come out, but the 24-hour IP browse my website page, can see in statistics to visit its footprint, and a more strange phenomenon is that there are three kinds of the IP source of pit, the first is to directly access, the second is through the bidding link, the third is the keywords from baidu search engine search strange XZBY SDFR S6YU such keywords entered - no matter what kind of the title, this web page you it search keywords are perfectly matched,
About treatment scheme is: through the server, the web firewalls, method such as.htaccess banned, but he couldn't help, a day in 1, 2, IP, banned after a day in 5 or 6 m, but in a couple of days and into 1, 2 m,
Have great god understand this aspect of the can contact,

CodePudding user response:

http://tool.chinaz.com/robots/try to use robots. TXT although don't know what do you use your server,

CodePudding user response:

Robots are not limiting the crawler? Can also be banned IP?

CodePudding user response:

refer to the second floor jiantu006 response:

,

robots is not limit the crawler? Can also be banned IP?

eliminate the crawler is brush flow, the WEB server should be able to set shielding IP, what also don't know what's your supplier system, or from the server Settings laid hands on him,

CodePudding user response:

DDOS attacks unanswered, defense is the only choice

CodePudding user response:

Can't, can only spend money flow cleaning, make the front to give you with the machine

CodePudding user response:

reference TinMan2012 reply: 3/f

Quote: refer to the second floor jiantu006 response:

Robots are not limiting the crawler? Can also be banned IP?

eliminate the crawler is brush flow, the WEB server should be able to set shielding IP, what also don't know what's your supplier system, or from the server Settings laid hands on him,

ali cloud server Linux system technology shielding the IP section, but I don't know what reason the IP for the three IP can also come in

CodePudding user response:

Don't know why your website is, but would it be the headline of the crawler

CodePudding user response:

Like the tide, whether ddos
First in the group policy, limit the IP try
Not have to spend some money, high server, cleaning, etc.
Treatment is generally difficult to yourself

CodePudding user response:

refer to 7th floor wearing a treble response:

don't know why your website is, but would it be the headline of the crawler

I don't know, bought another 50 m bandwidth

CodePudding user response:

Should be search engine spiders, set the robots file, set the scraping of the frequency, or install software fire drywall to block an IP

CodePudding user response:

Baidu inc., in some provinces and cities of subsidiary company (outside Beijing) some of the employees have repeatedly find we develop all kinds of creeper, to collect information, actually the rogue companies do this, there are also several abide by the rules of the crawler declared himself to be the crawler for your web server filter?

CodePudding user response:

Out of the thing, such as "was done by the unlicensed programmer", "was done by the third party company", all have already cancel the cooperation,