I want to scrap all the titles and descrpitions of the elements of a website using scrapy.

https://www.sitl.eu/fr-fr/exposants/liste-des-exposants.html

I've made the request in a scrapy shell, but I cannot understand why do I get an empty list for the titles and description;

page's inspect

>>> response.xpath('//div[@]/a[@h3]/text()').getall()
[]

CodePudding user response：

This xpath statement returns the list you are trying to get:

$x('//div[@]/div/a//text()')

After that, you will be able to get the single values if you iterate.

CodePudding user response：

Here is my python file, it's the list before extraction I can't get

import scrapy 
  
class sitl(scrapy.Spider): 
    name = "sitl"



def start_requests(self): 
    urls = [ 'https://www.sitl.eu/fr-fr/exposants/liste-des-exposants.html', ] 
      
    for url in urls: 
        yield scrapy.Request(url = url, callback = self.parse) 


def parse(self, response): 
      
    
      
    
    exposants = response.xpath('//div[@]/div/a//text()')    
      
    for exposant in exposants: 
        yield 
        { 
            'exposants': exposant 
        } 
          
        if 'sitl' in exposant:          
            yield scrapy.Request(url = exposant, callback = self.parse)