Hi so i want to scrape domain names and their prices but its returning null idk why

from bs4 import BeautifulSoup

url = 'https://www.brandbucket.com/styles/6-letter-domain-names?page=1'

response = requests.get(url)

soup = BeautifulSoup(response.text,'html.parser')
names = soup.findAll('div', {'class': "domainCardDetail"})
print(names)

CodePudding user response：

Check the status code of the response. When I tested there was 403 from the Web Server and because of that there is no such element like "domainCardDetail" div in response.

CodePudding user response：

The reason for this is that website is protected by Cloudflare. There are some advanced ways to bypass this.

The following solution is very simple if you do not need a mass amount of scraping. Otherwise, you may want to use "clouscraper" "Selenium" or another method to enable JavaScript on the website.

1. Open the developer console
2. Go to "Network". Make sure ticks are clicked as below picture. https://i.stack.imgur.com/v0KTv.png
3. Refresh the page.
4. Copy JSON result and parse it in Python https://i.stack.imgur.com/odX5S.png