CodePudding user response:
Because crawl site, different need to write different code itself, and the request of different parameters, to download the data,So, you need to figure out yourself to download data, need to pass any parameters, simulation, to write the code can be downloaded to,
Specific about how to write the crawler, you need to make clear the basic logic,
You can refer to my relevant series:
Crawl data: you want the crawler technology
How to written in Python crawler