Web Scraping using python and Beautiful soup-CodePudding

I'm trying to scrape this website and get the download links

https://gogohd.pro/download?id=OTk1OTk=&typesub=Gogoanime-SUB&title=Dragon Ball Super Episode 100

using this piece of code

# import libraries
from requests_html import HTMLSession

# specify the url
URL = 'https://gogohd.pro/download?id=MTkzNTU3&typesub=Gogoanime-SUB&title=Chainsaw Man Episode 1' 

session = HTMLSession()
r = session.get(URL)

for link in r.html.links:
    print(link)

But it's not returning the links and is instead returning it empty. I tried replicating it with selenium but to no avail :(

CodePudding user response：

Use the chrome option with following css selecto using selenium.

from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.common.by import By

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=chrome_options)
driver.get("https://gogohd.pro/download?id=MTkzNTU3&typesub=Gogoanime-SUB&title=Chainsaw Man Episode 1")
time.sleep(3)
downloadLinks =[link.get_attribute('href') for link in driver.find_elements(By.CSS_SELECTOR,"div.dowload>a")]
print(downloadLinks)
print(f"Download links count: {len(downloadLinks)}")

output:

['https://gogodownload.net/download.php?url=aHR0cHM6LyAdrefsdsdfwerFrefdsfrersfdsrfer363435349AawehyfcghysfdsDGDYdgdsfsdfwstdgdsgtertseWVpYnU0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjM2MHAubXA0P3Rva2VuPWtQaXpiR0xjQ2lDaXdJY25xNnNRSHcmZXhwaXJlcz0xNjcxNjM2ODQ2JmlkPTE5MzU1Nw==', 'https://gogodownload.net/download.php?url=aHR0cHM6LyAdeqwrwedffryretgsdFrsftrsvfsfsr9seWVpYnAawehyfcghysfdsDGDYdgdsfsdfwstdgdsgtertU0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjQ4MHAubXA0P3Rva2VuPWpRUzd1UnA4U2Z0X0tUeWYtRGNXc1EmZXhwaXJlcz0xNjcxNjM2ODQ2JmlkPTE5MzU1Nw==', 'https://gogodownload.net/download.php?url=aHR0cHM6LyAdrefsdsdfwerFrefdsfrersfdsrfer363435349AdeqwrwedffryretgsdFrsftrsvfsfsrseWVpYnU0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjcyMHAubXA0P3Rva2VuPVYxb1ZsZDI3VGtoZGdNRjVURS1yYmcmZXhwaXJlcz0xNjcxNjM2ODQ2JmlkPTE5MzU1Nw==', 'https://gogodownload.net/download.php?url=aHR0cHM6LyAdeqwrwedffryretgsdFrsftrsvfsfsr9seWVpYnAdrefsdsdfwerFrefdsfrersfdsrfer36343534U0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjEwODBwLm1wND90b2tlbj04clZFSUlOOGpicTYtZWx0bmNqT3VRJmV4cGlyZXM9MTY3MTYzNjg0NiZpZD0xOTM1NTc=', 'https://streamsss.net/d/aat4tegpjl4g', 'https://dood.wf/d/c2y9u6k23a2o', 'https://fembed9hd.com/f/gl1enu-757y2l2l', 'https://bodelen.com/afu.php?zoneid=2052717']
Download links count: 8