I wrote a climb with scrapy framework China weather net crawl, only inside the xpath, 3000 data used for 30 seconds,
Use BeautifulSoup wrote a script crawler, 3000 data only for 3 seconds,
In this gap is how cause?

CodePudding user response:

You can see scrapy - redis, distributed on the whole, and scrapy itself also can modify the parameters in the configuration