To climb in jilin province people's government websites, but found that channelID is random changes, as shown through parameter request for web information, the great god excuse me, how to deal with?

CodePudding user response:

Which page you want to climb? What is the page? What is the page source?

CodePudding user response:

The debug to see what is generated

CodePudding user response:

Has been solved, using selenium can climb out, is now thinking how to selenium and integrate the scrapy

CodePudding user response:

Can write the selenium in the spiders, or write middlerware

CodePudding user response:

reference 4 floor archaeologists lx response:

can write the selenium in the spiders, or write middlerware

uh-huh, I see many tutorials in setting, middlerware has changes inside, mentally, crawler small white to grope for the