Python Can not parse website Human problem but open in website-CodePudding

i try to parse website but there is error You need to enable support for <a href="https://yandex.ru/support/common/browsers-settings/browsers-java-js-settings.html">js</a> in your browser to visit this site

I try this code

import requests
from bs4 import BeautifulSoup

URL = "https://siteurl"
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}
page = requests.get(URL.strip(), headers=headers, timeout=100)

soup = BeautifulSoup(page.content, "html.parser")
print(soup.contents)

when i try to open in browser , it's work . Any solution?

CodePudding user response：

import requests
from bs4 import BeautifulSoup

URL = "https://siteurl"
page = requests.get(URL)

soup = BeautifulSoup(page.text, "html.parser")
print(soup)

CodePudding user response：

The website asks for the enabled JavaScript. BeatifulSoup does not mimick a full-fledged web-browser, so it lacks JavaScript functionality. You can try using Selenium BeatifulSoup together since Selenium behaves as a full fledged browser.