Home > front end >  I can't get the value of a href with webscrapping, the field comes "none"
I can't get the value of a href with webscrapping, the field comes "none"

Time：08-05

I'm trying to do a web scraping with scrapy, but when I try to get the text from the href, it comes as 'none', can someone help me?

I need to get the values ​​of the "sinonimos" classes

The page, the values ​​I want to get are these:

image

Code:

import scrapy

class SinonimoSpider(scrapy.Spider):
    name = 'sinonimo'
    start_urls = ['https://www.sinonimos.com.br/pedido/']
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.2840.71 Safari/539.36'

    def parse(self, response):
        for filmes in response.css('.sinonimo'):
            yield{
                'sinonimo': filmes.css('.sinonimo a::text').get()
               
            }

Result:

.........
........
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': None}
2022-08-04 00:23:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
........
........

CodePudding user response：

You are already looping for ".sinonimo" when you wrote for filmes in response.css('.sinonimo'):. It fails because you tried to do it again, and it also includes all the link tags that contains the words you need. Just a ::text is sufficient to get all the words.

import scrapy
import requests
from scrapy.selector import Selector

class SinonimoSpider(scrapy.Spider):
    name = 'sinonimo'
    start_urls = ['https://www.sinonimos.com.br/pedido/']
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.2840.71 Safari/539.36'}

    def get_url(self, url):
        return Selector(requests.get(url, headers=self.headers))

    def parse(self, response):
        for filmes in response.css('.sinonimo'):
            # get the text of all links with class 'sinonimo'
            yield{
                'sinonimo': filmes.css('::text').extract_first()

            }

test = SinonimoSpider()
for word in test.parse(test.get_url('https://www.sinonimos.com.br/pedido/')):
    print(word)

Output:

{'sinonimo': 'encomenda'}
{'sinonimo': 'pacote'}
{'sinonimo': 'volume'}
{'sinonimo': 'item'}
{'sinonimo': 'objeto'}
{'sinonimo': 'produto'}
{'sinonimo': 'unidade'}
{'sinonimo': 'embalagem'}
{'sinonimo': 'embrulho'}
{'sinonimo': 'carga'}
{'sinonimo': 'fardo'}
{'sinonimo': 'maço'}
{'sinonimo': 'rolo'}
{'sinonimo': 'bala'}
{'sinonimo': 'cartucho'}
{'sinonimo': 'envoltório'}
{'sinonimo': 'invólucro'}
{'sinonimo': 'solicitação'}
{'sinonimo': 'petição'}
{'sinonimo': 'requerimento'}
{'sinonimo': 'requisição'}
{'sinonimo': 'ordem'}
{'sinonimo': 'pedimento'}
{'sinonimo': 'reclamação'}
{'sinonimo': 'recurso'}
{'sinonimo': 'pretensão'}
{'sinonimo': 'demanda'}
{'sinonimo': 'exigência'}
{'sinonimo': 'reclamo'}
{'sinonimo': 'apelação'}
{'sinonimo': 'embargo'}
{'sinonimo': 'agravo'}
{'sinonimo': 'súplica'}
{'sinonimo': 'rogo'}
{'sinonimo': 'prece'}
{'sinonimo': 'oração'}
{'sinonimo': 'clamor'}
{'sinonimo': 'imploração'}
{'sinonimo': 'suplicação'}
{'sinonimo': 'invocação'}
{'sinonimo': 'peditório'}
{'sinonimo': 'rogação'}
{'sinonimo': 'instância'}
{'sinonimo': 'insistência'}
{'sinonimo': 'solicitado'}
{'sinonimo': 'requerido'}
{'sinonimo': 'necessitado'}
{'sinonimo': 'exigido'}
{'sinonimo': 'reivindicado'}
{'sinonimo': 'requisitado'}
{'sinonimo': 'rogado'}
{'sinonimo': 'suplicado'}
{'sinonimo': 'pleiteado'}
{'sinonimo': 'pretendido'}
{'sinonimo': 'demandado'}
{'sinonimo': 'reclamado'}
{'sinonimo': 'diligenciado'}

CodePudding user response：

Your code is actually really close. The only thing you need to change to make it right is to use a.sinonimo::text instead of .sinonimo a::text.

What you could also do, since the class sinonimo and tag a combo exclusively belong to the words you are trying to scrape, you can simplify it using a yield from expression.

For Example

import scrapy
class SinonimoSpider(scrapy.Spider):
    name = 'sinonimo'
    start_urls = ['https://www.sinonimos.com.br/pedido/']
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.2840.71 Safari/539.36'}

    def parse(self, response): yield from ({'sinonimo':i.get()} for i in response.css('a.sinonimo::text'))

OUTPUT

....
....
https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'encomenda'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'pacote'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'volume'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'item'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'objeto'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'produto'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'unidade'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'embalagem'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'embrulho'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'carga'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'fardo'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'maço'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'rolo'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'bala'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'cartucho'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'envoltório'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'invólucro'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'solicitação'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'petição'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'requerimento'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'requisição'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'ordem'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'pedimento'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'reclamação'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'recurso'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'pretensão'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'demanda'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'exigência'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'reclamo'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'apelação'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'embargo'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'agravo'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'súplica'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'rogo'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'prece'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'oração'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'clamor'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'imploração'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'suplicação'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'invocação'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'peditório'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'rogação'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'instância'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'insistência'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'solicitado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'requerido'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'necessitado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'exigido'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'reivindicado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'requisitado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'rogado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'suplicado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'pleiteado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'pretendido'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'demandado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'reclamado'}
2022-08-03 23:27:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sinonimos.com.br/pedido/>
{'sinonimo': 'diligenciado'}
...
...

Page link：https//www.codepudding.com/frontend/499288.html

Prev:How can I scrape the Ph elements of this list page?
Next:Jquery ui autocomplete trigger select event then ajax Post to call c# action, but it returns 400 err
  • Related

About Us:  Contact Us      Terms of Service       Privacy Policy

Copyright © 2010-2023，Powered By CodePudding