Home > other >  Python crawler
Python crawler

Time:09-18

When I am with scrapy crawler encountered the following problem, what reason is excuse me somebody know how to solve? (I can crawl before suddenly like this, and I can go in site)
C: \ Users \ \ Administrator \ \ AppData \ Local \ designed \ Python Python37 \ Python exe D:/DaiMa/DFCF/DFCF/start py
The 2020-06-07 12:50:47 [scrapy. Utils. Log] INFO: scrapy 1.7.3 started (bot: DFCF)
The 2020-06-07 12:50:47 [scrapy. Utils. Log] INFO: Versions: LXML 4.4.1.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.7.0, Python 3.7.2 (tags/v3.7.2:9 a3ffc0492, Dec 23, 2018, 23:09:28) [MSC v. 1916 64 - bit (AMD64)], pyOpenSSL 19.0.0 of 1.1.1 c 28 May 2019), cryptography Platform Windows - 2.7-6.1.7601 - SP1
INFO: the 2020-06-07 12:50:47 [scrapy. Crawler] Overridden Settings: {' BOT_NAME ':' DFCF ', 'DOWNLOAD_DELAY: 1.5,' NEWSPIDER_MODULE ':' DFCF. Spiders', 'SPIDER_MODULES: [' DFCF. Spiders']}
The 2020-06-07 12:50:48 [scrapy. Extensions. Telnet] INFO: Telnet Password: 63 a869a4bbe34455
INFO: the 2020-06-07 12:50:48 [scrapy. Middleware] Enabled extensions:
[' scrapy. Extensions. Corestats. Corestats',
'scrapy. Extensions. Telnet. TelnetConsole',
'scrapy. Extensions. Logstats. Logstats']
INFO: the 2020-06-07 12:50:50 [scrapy. Middleware] Enabled downloader middlewares:
[' DFCF. Middlewares. UserAgentDownloadMiddleware ',
'DFCF. Middlewares. IPProxyDownloadMiddleware',
'scrapy. Downloadermiddlewares. Httpauth HttpAuthMiddleware',
'scrapy. Downloadermiddlewares. Downloadtimeout DownloadTimeoutMiddleware',
'scrapy. Downloadermiddlewares. Defaultheaders DefaultHeadersMiddleware',
'scrapy. Downloadermiddlewares. Useragent. UserAgentMiddleware',
'scrapy. Downloadermiddlewares. Retry. RetryMiddleware',
'scrapy. Downloadermiddlewares. Redirect MetaRefreshMiddleware',
'scrapy. Downloadermiddlewares. Httpcompression HttpCompressionMiddleware',
'scrapy. Downloadermiddlewares. Redirect RedirectMiddleware',
'scrapy. Downloadermiddlewares. Cookies, CookiesMiddleware',
'scrapy. Downloadermiddlewares. Httpproxy HttpProxyMiddleware',
'scrapy. Downloadermiddlewares. Stats. DownloaderStats']
INFO: the 2020-06-07 12:50:50 [scrapy. Middleware] Enabled spiders middlewares:
[' scrapy. Spidermiddlewares. Httperror. HttpErrorMiddleware ',
'scrapy. Spidermiddlewares. Can use OffsiteMiddleware',
Scrapy. Spi '
  • Related