Home > OS >  Web Scraping-Pairing asscoiated prices including real price
Web Scraping-Pairing asscoiated prices including real price

Time:10-27

I am trying to scrape this website enter image description here


But I recommend to save the prices under Store Names, not under "price_XY":

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://99petshops.com.au/Search?brandName=Ziwi Peak&animalCode=DOG&storeId=89/&page=1"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

all_info = []
for item in soup.select(".pd-info"):
    title = item.h2.get_text(strip=True)
    price = item.select_one('span:-soup-contains("Price")').span.text
    d = {"Title": title, "Price": price}

    for p in item.select(".sp-price"):
        d[p.find_next("img")["alt"]] = p.get_text(strip=True)

    all_info.append(d)

df = pd.DataFrame(all_info).fillna("")
print(df)

df.to_csv("data.csv", index=False)

Prints:

                                                      Title    Price VetShopAustralia Vet Products Direct Pet Shop Direct    Petso  PetPost Pet Chemist Your PetPA Pet Circle  Petbarn World for Pets Lucky Pet  Stefmar Budget Pet Products Best Friends Pets Kellyville Pets iPetStore My Pet Warehouse Pet City Pets Unleashed     PetO PETstock Pet House Habitat Pets Pet Culture Peticular Vet n Pet Direct   Petzoo Woonona Pet Foods Bundi Pet Pets on the Park Mega Pet Warehouse     eBay Petceutics The Animal Pharmacy Pacific Pet Supplies Sierra Pet Products Woofers World Discount Pet Meds
0     Ziwi Peak Dog Air-Dried Free Range Chicken Recipe 1Kg   $57.75           $64.60              $64.60          $64.95   $64.95   $64.99      $66.29     $67.32     $69.69   $69.69         $69.95    $70.19   $71.99              $72.99            $73.39          $75.15    $75.95           $76.99   $77.99         $77.99   $79.99   $81.54    $81.99       $83.49      $83.49    $83.50                                                                                                                                                                                                           
1   Ziwi Peak Dog Air-Dried Free Range Chicken Recipe 2.5Kg  $127.80                                                       $140.65  $141.68     $140.24    $147.84    $152.14  $152.14        $149.95   $148.49  $166.99             $159.99           $162.90         $159.30   $155.95          $167.00  $164.99        $164.99  $169.99  $181.00   $176.99      $176.99     $176.99   $176.95          $138.95  $144.99           $159.95   $164.99                                                                                                                                                     
2                 Ziwi Peak Dog Air-Dried Lamb Recipe 2.5Kg  $127.80                                                       $140.65  $140.63     $140.24    $147.84    $159.99  $159.99        $149.95   $148.49  $155.99             $159.99           $162.90         $159.30   $155.95          $165.00                 $164.99  $169.99  $181.00   $174.99      $176.99     $176.99   $176.95          $138.95  $144.99                     $164.99          $160.15                                                                                                                                    
3                  Ziwi Peak Dog Air-Dried Beef Recipe 454g   $29.25                                                                 $34.99      $36.13     $34.32     $35.99   $39.09         $37.95    $38.25   $38.99              $35.99            $40.28          $40.95    $38.95           $41.99   $42.50         $42.50   $44.99   $44.75    $42.49       $45.49      $45.49    $45.50           $32.95                     $35.95                     $37.20             $43.99                                                                                                                 
4    Ziwi Peak Dog Air-Dried Free Range Chicken Recipe 454g   $29.25                                               $34.95                        $36.13     $34.32     $39.09   $39.09         $37.95    $38.25   $36.99              $37.99            $40.28          $40.95    $38.95           $41.99   $42.50         $42.50   $44.99             $42.49       $45.49      $45.49    $45.50           $32.95                     $35.95                     $35.00             $43.99                                                                                                                 
5     Ziwi Peak Dog Air-Dried Free Range Chicken Recipe 4Kg  $216.22          $222.75             $222.75         $219.95           $223.61     $220.99    $233.64    $230.23  $238.99        $228.85   $233.99  $244.99             $249.99           $257.40         $251.10                    $261.00  $259.99        $259.99  $279.99  $286.00   $259.99      

...
  • Related