Home > front end >  Scraping href links in Python using BeautifulSoup
Scraping href links in Python using BeautifulSoup

Time:10-29

I am trying to scrape the links that are stored in the href values from this website: Here is a photo of the links

I tried the following code:

url = "https://www.fotmob.com/teams/8676/fixtures/wycombe-wanderers?page=1"
r = requests.get(url)
html_doc = r.content
soup = BeautifulSoup(html_doc, 'html.parser')

[tag['href'] for tag in soup.find_all('a',{'class':'css-11smhdq-FtContainer e1ym2d3s2'})]

But this just returns an empty list. Does anyone know how to fix this?

CodePudding user response:

The required data is loaded dynamically by JavaScript that's why you are getting empty output because bs4 can't render JS but they are using API and you can extract data from API url instead.

import requests

api_url= 'https://www.fotmob.com/api/teams?id=8676&timezone=Asia/Dhaka&ccode3=BGD'
headers={
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }
data=[]

res=requests.get(api_url,headers=headers)
#print(res)
for item in res.json()['fixtures']['allFixtures']['fixtures']:
    url = item['pageUrl']
    data.append({'url':'https://www.fotmob.com'   url})
print(data)

Output:

[{'url': 'https://www.fotmob.com/match/3915857/matchfacts/wycombe-wanderers-vs-burton-albion'}, {'url': 'https://www.fotmob.com/match/3915859/matchfacts/bolton-wanderers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3917827/matchfacts/northampton-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915875/matchfacts/wycombe-wanderers-vs-shrewsbury-town'}, {'url': 'https://www.fotmob.com/match/3915880/matchfacts/exeter-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915866/matchfacts/barnsley-vs-wycombe-wanderers'}, {'url': 
'https://www.fotmob.com/match/3992753/matchfacts/wycombe-wanderers-vs-bristol-city'}, {'url': 'https://www.fotmob.com/match/3915922/matchfacts/wycombe-wanderers-vs-charlton-athletic'}, {'url': 'https://www.fotmob.com/match/3940648/matchfacts/wycombe-wanderers-vs-tottenham-hotspur-academy'}, {'url': 'https://www.fotmob.com/match/3915928/matchfacts/fleetwood-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915936/matchfacts/wycombe-wanderers-vs-accrington-stanley'}, {'url': 'https://www.fotmob.com/match/3915957/matchfacts/derby-county-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3940645/matchfacts/stevenage-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915945/matchfacts/sheffield-wednesday-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915987/matchfacts/wycombe-wanderers-vs-plymouth-argyle'}, {'url': 'https://www.fotmob.com/match/3915978/matchfacts/oxford-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915997/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3940637/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3916030/matchfacts/milton-keynes-dons-vs-wycombe-wanderers'}, {'url': 
'https://www.fotmob.com/match/3916003/matchfacts/wycombe-wanderers-vs-cambridge-united'}, {'url': 'https://www.fotmob.com/match/3916015/matchfacts/wycombe-wanderers-vs-morecambe'}, {'url': 'https://www.fotmob.com/match/3915901/matchfacts/wycombe-wanderers-vs-port-vale'}, {'url': 'https://www.fotmob.com/match/4050422/matchfacts/wycombe-wanderers-vs-walsall'}, {'url': 'https://www.fotmob.com/match/3916041/matchfacts/forest-green-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916064/matchfacts/cheltenham-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916055/matchfacts/wycombe-wanderers-vs-portsmouth'}, {'url': 'https://www.fotmob.com/match/3916098/matchfacts/lincoln-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916093/matchfacts/wycombe-wanderers-vs-ipswich-town'}, {'url': 'https://www.fotmob.com/match/3916109/matchfacts/wycombe-wanderers-vs-bristol-rovers'}, {'url': 'https://www.fotmob.com/match/3916141/matchfacts/plymouth-argyle-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916154/matchfacts/peterborough-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916114/matchfacts/wycombe-wanderers-vs-oxford-united'}, {'url': 'https://www.fotmob.com/match/3916126/matchfacts/wycombe-wanderers-vs-sheffield-wednesday'}, {'url': 'https://www.fotmob.com/match/3916124/matchfacts/bristol-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916164/matchfacts/wycombe-wanderers-vs-fleetwood-town'}, {'url': 'https://www.fotmob.com/match/3916202/matchfacts/port-vale-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916186/matchfacts/wycombe-wanderers-vs-derby-county'}, {'url': 'https://www.fotmob.com/match/3916187/matchfacts/accrington-stanley-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916247/matchfacts/wycombe-wanderers-vs-bolton-wanderers'}, {'url': 'https://www.fotmob.com/match/3916223/matchfacts/shrewsbury-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916231/matchfacts/wycombe-wanderers-vs-exeter-city'}, {'url': 'https://www.fotmob.com/match/3916228/matchfacts/burton-albion-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916285/matchfacts/wycombe-wanderers-vs-barnsley'}, {'url': 'https://www.fotmob.com/match/3916279/matchfacts/charlton-athletic-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916314/matchfacts/wycombe-wanderers-vs-milton-keynes-dons'}, {'url': 'https://www.fotmob.com/match/3916321/matchfacts/ipswich-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916303/matchfacts/wycombe-wanderers-vs-forest-green-rovers'}, {'url': 'https://www.fotmob.com/match/3916358/matchfacts/morecambe-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916335/matchfacts/cambridge-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916349/matchfacts/wycombe-wanderers-vs-lincoln-city'}, {'url': 'https://www.fotmob.com/match/3916377/matchfacts/wycombe-wanderers-vs-cheltenham-town'}, {'url': 'https://www.fotmob.com/match/3916378/matchfacts/portsmouth-vs-wycombe-wanderers'}]

screenshot

  • Related