Beautiful Soup scraper gives "Access Denied" even though user-agent string is specified-CodePudding

I am trying to scrape this website: https://www.ralphlauren.com/men?webcat=men but I don't get the HTML of the link, instead I get an HTML which states Access to this page has been denied.

I tried using user agent string in the header as suggested here solution: Scraper in Python gives "Access Denied"

but I still get the same error.

Current Output:

  <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<title>Access to this page has been denied.</title>
<link href="https://fonts.googleapis.com/css?family=Open Sans:300" rel="stylesheet"/>

My Code:

from bs4 import BeautifulSoup as soup
import requests
url = "https://www.ralphlauren.com/men?webcat=men"

def make_soup(url):
  header = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"}
  page = requests.get(url,headers=header)
  page_soup = soup(page.content, 'lxml')
  return page_soup 

print(make_soup(url))

CodePudding user response：

The website is in cloudflare protection. You can use cloudscraper in lieu of requests

from bs4 import BeautifulSoup
import cloudscraper
scraper = cloudscraper.create_scraper(delay=10,   browser={'custom': 'ScraperBot/1.0',})
url = 'https://www.ralphlauren.com/men?webcat=men'
req = scraper.get(url)
print(req)
soup = BeautifulSoup(req.text,'lxml')

Output:

<Response [200]>