Ebay Scraper code most of the time doesn't work-CodePudding

I'm currently following a tutorial to create an Ebay Scraper from the following link:

https://www.youtube.com/watch?v=csj1RoLTMIA&t=290s

I'm mid-way through the code and suddenly notice that my code below only works sometimes:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=gaggia classic&_sacat=0&LH_TitleDesc=0&LH_Auction=1&LH_Sold=1&rt=nc'

def get_data(url):
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    return soup

def parse(soup):
    results = soup.find_all('div', {'class': 's-item__info clearfix'})
    print(len(results))
    return

soup = get_data(url)
parse(soup)

Most of the time the code spits out 0 but sometimes it spits out 51 (Which is the correct answer). Anybody knows how to make it work?

CodePudding user response：

The video is over a year old, and big companies like EBay are taking action to prevent things like data miners and web scrapers. You may be getting rate limited (sending requests too fast and flagging anti-bot systems).

CodePudding user response：

This article from zyte (formerly ScrapingHub) might help you - How to scrape the web without getting blocked