Scrapy / xpath not working - only with css-CodePudding

i try to scrape this site: https://www.magnatiles.com/products/page/1/

I get all the elements with:

products = response.xpath("//ul[@class='products']//ancestor::li")

No i try to find the price for all elements in the scrapy shell - at first i tried:

>>> for p in products:
...  p.xpath("//span[@class='price']//child::bdi/text()").get()           
... 
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'

It seems that with that i only get the first entry altough i am using the loop

Then i tried it with css-selecting and it worked:

>>> for p in products:
...  p.css("span.price bdi::text").get()
... 
'134.99'
'49.99'
'39.99'
'39.99'
'39.99'
'129.99'
'24.99'
'49.99'
'119.99'

Why is this not working when i am using the xpath-selector?

CodePudding user response：

After iterating to select xpath you have to use .// to get the desired result. Try as follows:

p.xpath(".//span[@class='price']//child::bdi/text()").get()

CodePudding user response：

If you want to use a xpath on already defined web element

then you can use

>>> for p in products:
...  p.xpath(".//span[@class='price']//child::bdi/text()").get()

CodePudding user response：

This xpath selector works in Chrome too.

//span[@]//child::bdi/text()