How to keep logged in on a website using requests library (Python)?-CodePudding

I have a code that allowed me to web-scrap a website which requires to log in.

I used to log in using Selenium, then get the cookies and put them into a requests.session() like this :

#Install and use the last version of ChromeDriver

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

#Log in to the website

driver.get(url)
driver.find_element_by_name('username').send_keys(e_mail)
driver.find_element_by_name('password').send_keys(mdp)
driver.find_element_by_name('Submit').send_keys('')
maybe_later_css = 'button[]'
driver.find_element_by_css_selector(maybe_later_css).click()

#Get the cookies after being logged in

cookies = driver.get_cookies()

#Put the cookies into a requests.session()

with requests.Session() as s:
for cookie in cookies:
    s.cookies.set(cookie['name'], cookie['value'])

This code worked perfectly for a year but suddenly it didn't anymore. Indeed, when I try a get request on that website, it appears that I'm not logged in (while it worked before).

I tried to add some headers like User-Agent and others, but without success so far.

Here is the website : https://tennis.paris.fr/tennis/jsp/site/Portal.jsp?page=recherche&view=recherche_creneau#!

If anyone has an idea or a solution to fix this, I would be really grateful.

CodePudding user response：

I was playing with your code and added some lines, and I got something like this:

#Install and use the last version of ChromeDriver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

#Log in to the website
driver.maximize_window()
driver.get("https://tennis.paris.fr/tennis/jsp/site/Portal.jsp?page=recherche&view=recherche_creneau#!")
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "banner-mon-compte__connexion-wrapper")))
driver.find_element(By.CLASS_NAME, "banner-mon-compte__connexion-wrapper").click()
driver.switch_to.window(driver.window_handles[1])
time.sleep(2.0)
driver.find_element(By.NAME, "username").send_keys(e_mail)
driver.find_element(By.NAME, "password").send_keys(mdp)
driver.find_element(By.NAME, "Submit").click()
time.sleep(1.0)
driver.switch_to.window(driver.window_handles[0])

#Get the cookies after being logged in
cookies = driver.get_cookies()
print(cookies)

#Put the cookies into a requests.session()
s = requests.Session()
for cookie in cookies:
    s.cookies.set(cookie['name'], cookie['value'])

print(cookie)

Sorry for amateur automation coding with time.sleep(), wanted to finish this quickly, now you need to switch from window to window when you want to log in to the website.