I am receiving this 504 gateway error while using splash with scrapy while learning splash where I was trying to crawl this

CodePudding user response：

The url you are trying to scrape takes long to load. Even if you try out in the browser, you will note that it takes time to fully load and stop spinning.

Splash therefore times out before the page is fully loaded and returned.

You need to do two things.

First increase the max timeout value when starting the splash server like below.

docker run -p 8050:8050 scrapinghub/splash --max-timeout 3600

Second, in the spider, you can provide a timeout value which is less than or equal to the max-timeout value of the splash server.

yield SplashRequest(url=url, args={"timeout": 3000})