Python Scrapy Loop Through The Same Urls-CodePudding

I'm trying to use scrapy to go through a couple of different urls but the problem is that I don't want to go through them once. I want to be able to loop through the same urls "Forever". The code below is what I have so far but it doesn't work and I have no clue why.

        while True:
            looptime = time.time()   900
            while time.time() < looptime:
                for word in words:
                    url = 'https://myurl.com
                    yield scrapy.Request(url=url, callback=self.parse)
            dosomething()

Let's say I have 2 links, one to google and one to youtube. Currently, my code would only go through them once and then finish the script, but the desired result is to keep looping through the same 2 links all the time. So it would go through Google, Then Youtube, Google, Youtube and so on. Thank you!

CodePudding user response：

Add dont_filter=True to the request to prevent scrapy from filtering duplicate requests.

See this