Edit: Got the answer. The autothrottle was limited to default. Now I need to limit it to the number of website pages. The code looks like this CONCURRENT_REQUESTS = 3

I am using scrapy playwright. I want to loop through some sites. The scrapy spider will request to extract data through each website link. After rendering through each request, do I need to close the tab? like using this:

def parse(self, response):
page = response.meta["playwright_page"]
await page.close()

CodePudding user response：

Edit: Got the answer. The autothrottle was limited to default. Now I need to limit it to the number of website pages. The code looks like this CONCURRENT_REQUESTS = 3