Just learning scrapy, now encountered this problem is bothering me, is this, this is my program from the parse into the call after the next_one function, and then after entering next_one formrequest request, the results back to transfer into next_two, after entering next_two next_three function called again, but this time just can not enter the next_three function, can not enter the debug, and then switch to request the requested page, and the debug, this time will be able to enter the next_three function, what is this, to request the same formrequest is an asynchronous callback, why can a complete callback, a problem that can not be so I have a few 1, why the parse invokes the function also needs to use yield, need what to scrapy
2, why the same program can request into the fourth function, and the formrequest enter
3, what scrapy should be how to call the common function correctly, such as in the parse correctly call other ordinary function trouble everyone help me,

CodePudding user response:

CodePudding user response:

Can only use yield to call a function, do not know why,,,

CodePudding user response:

I also encountered this problem, can't direct call spiders internal functions,
My solution is:
Put the need to take to the method of external, spiders can call by function, defined function or use the class can be directly,
But external functions not directly use yield to return to the item, return to get the data first, inside the spiders use yield again can,

 
. 
The class MySpider (scrapy. Spiders) : 
Def parse (self, response) : 
. 
Yield base. The info (response) 
. 
The class Base: 
Def info (self, response) : 
. 
Myitem=myitem () 
. 
Return myitem 

The

reference 3 floor notice last reply:

can only use yield to call a function, do not know why,,,

Yield within the spiders can only apply the Request, the item, dict, etc., misuse has an error prompt,

CodePudding user response:

Actually you can call, is my wrong

CodePudding user response:

The

reference 5 floor notice last reply:

actually can call, I made a mistake

It was so, that's amazing, isn't there must be return? Call must receive to return to a successful call