Home > other >  Python crawl data every 15 pages, behind of the content is empty.
Python crawl data every 15 pages, behind of the content is empty.

Time:05-13

I crawl the data, a url web access head headers included in the content - length: 16 attributes, when circulating crawl data, can only climb the previous 15 pages, after 16 pages of the content is empty, if the content - length: 16 attribute limits the 15 times, each time can only access I thought of a way, every visit to 15 times to manually rewrite cycle, visit again, but this manual is very troublesome, don't know whether there is a break limit method?
  • Related