InvalidURL: Failed to parse: <Response [403]>-CodePudding

I tried scraping a website (Futbin) with BeatifulSoup and Requests but when I try to run my code I get this error:

InvalidURL: Failed to parse: <Response [403]>

A way how to fix this problem will be appreciated.

The code that I used:

import requests
url = requests.get("https://www.futbin.com/23/player/50188/bruno-guimaraes")
html_doc = url.content
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, "html.parser")

response = requests.get(url)
htmlText = response.text
part1 = htmlText.split ('')                               
part2 = part1.split ("price_big_right")[1]
part3 = part2.split (">")[2]
part4 = part3.split ("<")[0]
part5 = part4.replace (",",".")
wert = float (part5)
wert

I expected to get the value from the class above as a float.

CodePudding user response：

This response is because the page is blocking your request. It's a way to handle web scrapping. To avoid this answer you have to aggreate headers a your request.

import requests
headers = {'User-Agent':"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"}
res = requests.get('https://www.futbin.com/23/player/50188/bruno-guimaraes',headers=headers)

But, investigating the page i realized the page don´t send a respone with the prices but waits for the page to load so that it loads the prices through a script. Then when you go to extract the price return character '-'. It is a part of the html that is returned by python

<span >
    <span id="ps-lowest-1">-</span>
</span>

I recommend use selenium library in this case. I leave you some links to learn about selenium.

So Its very impossible to send requests to cloudflare protected sites.