Unable to scrape description of webpage in Scrapy-CodePudding

I want scrape data using Scrapy and here is the link

I use this code to get the Description

'description': response.css('#tab-description p::text').extract(),

But the response was

description': ['    ', 'None  ', '  ', 'Unisex  ', '  ', '12-15 Years  ', '  ', 'Grownups  ', '  ', 'Paper  ', '  ', 'Landscape  ', '  ', 'SMW783   ']

It ignores <strong> and <br> tags.

I need the output like this

<p>    <strong>Brand Name: </strong>None  <br>  <strong>Gender: </strong>Unisex  <br>  <strong>Age Range: </strong>12-15 Years  <br>  <strong>Age Range: </strong>Grownups  <br>  <strong>Material: </strong>Paper  <br>  <strong>Style: </strong>Landscape  <br>  <strong>Model Number: </strong>SMW783   </p>

CodePudding user response：

You can try using xpath:

I tested this and it seems to work:

for element in response.xpath("//div[@id='tab-description']/p"):
    values = element.xpath("./text()").getall()
    labels = element.xpath("./strong/text()").getall()
    values = [i for i in values if i.strip()]
    data = {labels[i]:values[i] for i in range(len(labels))}
    print(data)

output: {'Brand Name: ': 'None ', 'Gender: ': 'Unisex ', 'Age Range: ': 'Grownups ', 'Material: ': 'Paper ', 'Style: ': 'Landscape ', 'Model Number: ': 'SMW783 '}

This is just the html:

a = response.xpath("//div[@id='tab-description']/p").get()
print(a)

output

<p>    <strong>Brand Name: </strong>None  <br>  <strong>Gender: </strong>Unisex  <br>  <strong>Age Range: </strong>12-15 Years  <br>  <strong>Age Range: </strong>Grownups  <br>  <strong>Material: </strong>Paper  <br>  <strong>S
tyle: </strong>Landscape  <br>  <strong>Model Number: </strong>SMW783   </p>

CodePudding user response：

When /text() in XPath or ::text in CSS fails to produce the desired result, I use another library.

to install it.

pip3 install html2text

example

from html2text import HTML2Text
h = HTML2Text()
h.ignore_links = True
h.ignore_images = True
h.ignore_emphasis = True

#Inside the scrapy project
'description': h.handle(response.css('#tab-description p').get()).strip()

yield ....

CodePudding user response：

I tried this and it work

import scrapy
from bs4 import BeautifulSoup

class Google(scrapy.Spider):
    name = 'google'
    start_urls = ['https://bbdealz.com/product/funny-sports-game-2m-3m-4m-5m-6m-diameter-outdoor-rainbow-umbrella-parachute-toy-jump-sack-ballute-play-game-mat-toy-kids-gift/',]
    def parse(self, response):
        soup = BeautifulSoup(response.body, 'html.parser')
        yield {
        'title': response.css('h1::text').get(),
        'description': soup.select_one('#tab-description').select_one('p'),
        }

Regards