How do I retrieve the product title and price from the first Amazon result?-CodePudding

I am trying to create a Web Scraper using Beautiful Soup. As mentioned in the title, using Amazon, I am having difficulties with retrieving the product name and price from the first result.

Here's my code:

import requests
from bs4 import BeautifulSoup
def submit_search():
    product_name = product_entry.get()
    product_page = requests.get(f"https://www.amazon.com/s?k={product_name}")
    soup = BeautifulSoup(product_page.content, 'html.parser')

    price_element = soup.find(class_='a-price')
    if price_element is not None:
        price = float(price_element.text[1:])
        print(f"The price of the product is: ${price}")
    else:
        print("Could not find the price of the product")

An example output would be: User enters "Apple" --> prints the price of the first result of Apple on Amazon. I'm pretty sure it has something to do with the class, but I'm not sure. If someone could assist me, that would be greatly appreciated.

CodePudding user response：

You need to add the header to your request to pretend to be a browser - as is shown here.

I've changed your code as follows:

import requests
from bs4 import BeautifulSoup

def submit_search():

  headers = {
        "User-agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36",
        "Referer": "https://www.amazon.com"
  }
  
  product_name = input("Enter product name: ")
  print(f"https://www.amazon.com/s?k={product_name}")
  product_page = requests.get(f"https://www.amazon.com/s?k={product_name}", headers=headers)
  soup = BeautifulSoup(product_page.content, 'html.parser')
  
  price_element = soup.find_all("span", class_="a-offscreen")

  if price_element is not None:
      price = price_element[0].get_text("", strip=True)
      print(f"The price of the product is: {price}")
  else:
      print("Could not find the price of the product")

Result:

Enter product name: apple
https://www.amazon.com/s?k=apple
The price of the product is: $25.00

CodePudding user response：

To get a response from Amazon, you need to add the correct headers, otherwise, the page thinks that you a bot and will block you.

Also, to convert the price to a float, you should use:

price = float(price_element.text.split("$")[-1])

otherwise, youll get this as output:

'746.79$746.79'

and hence you won't be able to convert it to a float since it isn't a valid type.

So, try:

import requests
from bs4 import BeautifulSoup


def submit_search(product_name):
    HEADERS = {
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
        "referer": "https://www.amazon.com/",
    }
    product_page = requests.get(
        f"https://www.amazon.com/s?k={product_name}", headers=HEADERS
    )
    soup = BeautifulSoup(product_page.content, "html.parser")
    price_element = soup.find(class_="a-price")
    if price_element is not None:

        price = float(price_element.text.split("$")[-1])
        print(f"The price of the product is: ${price}")
    else:
        print("Could not find the price of the product")


submit_search("iphone")

Additionally, to get all the prices, you can try:

def submit_search(product_name):
    HEADERS = {
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
        "referer": "https://www.amazon.com/",
    }
    product_page = requests.get(
        f"https://www.amazon.com/s?k={product_name}", headers=HEADERS
    )
    soup = BeautifulSoup(product_page.content, "html.parser")
    for product in soup.select('.a-section.a-spacing-base'):
        title = product.find("h2").text
        price = product.select_one(".a-price-whole,.a-offscreen").text
        print("{}\t\t\t{}".format(price, title))

Prints (truncated):

$454.00         Apple iPhone 12, 128GB, Black - Fully Unlocked (Renewed)  
$429.99         Apple iPhone 12 Mini, 64GB, Red - Unlocked (Renewed Premium)  
$432.00         Apple iPhone 12 Mini, 64GB, Blue - Unlocked (Renewed Premium)  
$617.00         Apple iPhone 12, 256GB, Green - Unlocked (Renewed Premium)  
$593.75         Apple iPhone 13, 128GB, Blue - Unlocked (Renewed)

Note, Amazon has a system for detecting bots, so you might still not get an output, hence, you should use a try/except to see wheter you get an output, as you'll get an AttributeError.