The crypto exchange hotbit.io disabled some important endpoints I was using. Now I have to use the Cloudflare protected ones. My Idea was to solve Cloudflare with selenium/undetected_chromedriver and pass the cookies to my session.

Code:

import time

import requests
import undetected_chromedriver as uc

#create a browser and solve cloudflare
driver = uc.Chrome()
url = "https://www.hotbit.io/"
driver.get(url)
time.sleep(5)

#storing the cookies generated by the browser
request_cookies_browser = driver.get_cookies()
print(request_cookies_browser)

s = requests.Session()

#passing the cookies generated from the browser to the session
c = [s.cookies.set(c['name'], c['value']) for c in request_cookies_browser]

print(s.cookies)

resp = s.get(url)
print(resp.text)

I changed some values so I'm not leaking anything

Output: Cookies: https://pastebin.com/u7sGvjae html-hotbit: https://pastebin.com/6sAfhWtv

It looks like all cookies are set correctly but I'm still on the Cloudflare solving page with my session. Anyone has a solution to "bypass" the Cloudflare page and access the website with requests?

CodePudding user response：

Looks like cookies AND user-agent are needed. To be precise only the cf_clearance cookie AND user-agent are needed.

Solution would looks something like that:

import time
import undetected_chromedriver as uc
import requests

#create driver with selenium/undetected_chromedriver
driver = uc.Chrome()
driver.get("https://www.hotbit.io/")
time.sleep(5) #5s sleep, so the driver can solve cloudflare

#get cookies and user_agent
brCookies = driver.get_cookies()
ua = driver.execute_script("return navigator.userAgent")

#filter out the cf_clearance cookie
cf_cookies = [cookie for cookie in brCookies if cookie['name'] == 'cf_clearance'][0]['value']
cookies = {"cf_clearance": cf_cookies}
headers = {"user-agent": ua}

#create session and get your site
session = requests.session()
res = session.get("https://www.hotbit.io/", headers=headers, cookies=cookies)