Python Request with Cookies - Content blocked by Cookie Banner-CodePudding

I'm trying to access a website via Python Requests. To avoid the iframe of the "Cookie Banner" I want to pass the cookie that handles the banner.

With Selenium I already managed to figure out which cookie that is and there it works fine with just passing the key/value" pair. I already found online, that it is necessary to "get" the page before passing and then refreshing it with "get" again after adding the cookies.

website = "https://www.myfitnesspal.com"
path =  "path/to/your/chromedriver.exe"
service = Service(executable_path=path)
driver = webdriver.Chrome(service = service)

driver.get(website)
driver.add_cookie({'name': 'notice_preferences','value': '2:'})
driver.get(website)

So far so good. However, if I pass the same cookie that already worked in Selenium to a Python Request, the response.text that I receive still shows the content of the iframe and "Cookie Banner".

response = requests.get(website, cookies={"notice_preferences":"2:"})

Does anyone know why this is happening or if there is even a solution for this?

CodePudding user response：

I don't think that page's content is blocked by cookie banner. It's rather blocked by the lack of a proper user-agent in header. The following code will return the page content as seen in browser:

import requests
import pandas as pd


s = requests.Session()

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

url = 'https://www.myfitnesspal.com'

s.headers.update(headers)
s.cookies.set("name", "notice_preferences", domain="www.myfitnesspal.com/")
s.cookies.set("value", "2:", domain="www.myfitnesspal.com/")
r = s.get(url)
# print(r.text)
print(s.cookies)

Result printed in terminal:

<RequestsCookieJar[<Cookie split-id=e28e4968-c2e3-4145-9226-0d9db15bcffe for www.myfitnesspal.com/>, <Cookie name=notice_preferences for www.myfitnesspal.com//>, <Cookie value=2: for www.myfitnesspal.com//>]>

You can then navigate to another page in that website, and requests' Session will preserve the headers and cookies. Also, print out the text response, see if the info you're looking for is there. For more info on Requests, you can visit https://requests.readthedocs.io/en/latest/

EDIT: This is an a la carte XY problem - luckily the OP clarified it in comments. That data is being pulled via an XHR call from an API endpoint. To get the info youi want, you need to scrape the endpoint. THis is how you do it (after inspecting Dev Tools - Network Tab and seeing that endpoint):

import requests
import pandas as pd


s = requests.Session()

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

url = 'https://www.myfitnesspal.com/api/nutrition?query=banane&page=1&offset=10'

r = s.get(url)

df = pd.DataFrame(r.json()['items'])
display(df)

This will print in terminal:

item    tags    type
0   {'country_code': 'US', 'deleted': False, 'description': 'Banane', 'id': 1873022840, 'nutritional_contents': {'calcium': 0.5, 'carbohydrates': 22.84, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.33, 'fiber': 2.6, 'iron': 1.44444, 'monounsaturated_fat': 0.032, 'polyunsaturated_fat': 0.073, 'potassium': 358, 'protein': 1.09, 'saturated_fat': 0.112, 'sodium': 1, 'sugar': 12.23, 'trans_fat': 0, 'vitamin_a': 4.26667, 'vitamin_c': 14.5, 'vitamin_d': 0}, 'public': True, 'serving_sizes': [{'id': '67628178485117', 'index': 0, 'nutrition_multiplier': 1.18, 'unit': 'medium', 'value': 1}, {'id': '67078422671357', 'index': 1, 'nutrition_multiplier': 1.36, 'unit': 'large', 'value': 1}, {'id': '67628178485245', 'index': 2, 'nutrition_multiplier': 1.5, 'unit': 'cup, sliced', 'value': 1}, {'id': '67078414315389', 'index': 3, 'nutrition_multiplier': 2.25, 'unit': 'cup, mashed', 'value': 1}, {'id': '67628170129277', 'index': 4, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '67078414315517', 'index': 5, 'nutrition_multiplier': 0.283495, 'unit': 'oz', 'value': 1}, {'id': '67628170129405', 'index': 6, 'nutrition_multiplier': 0.81, 'unit': 'extra small', 'value': 1}, {'id': '67078422703997', 'index': 7, 'nutrition_multiplier': 1.52, 'unit': 'extra large', 'value': 1}, {'id': '67628178517885', 'index': 8, 'nutrition_multiplier': 4.53592, 'unit': 'lb(s)', 'value': 1}, {'id': '67078422704125', 'index': 9, 'nutrition_multiplier': 1e-05, 'unit': 'mg(s)', 'value': 1}, {'id': '67628178518013', 'index': 10, 'nutrition_multiplier': 10, 'unit': 'kg(s)', 'value': 1}, {'id': '67076304547197', 'index': 11, 'nutrition_multiplier': 0.00625, 'unit': 'mL, sliced ', 'value': 1}, {'id': '67626060361085', 'index': 12, 'nutrition_multiplier': 0.009375, 'unit': 'mL, mashed ', 'value': 1}, {'id': '67076304547325', 'index': 13, 'nutrition_multiplier': 6.25, 'unit': 'liter(s), sliced ', 'value': 1}, {'id': '67626060361213', 'index': 14, 'nutrition_multiplier': 9.375, 'unit': 'liter(s), mashed ', 'value': 1}], 'type': 'food', 'user_id': '133476501057389', 'verified': True, 'version': '199432263862133'}   [canonical, best_match] food
1   {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Une banane', 'id': 2007191148, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 27, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 105}, 'fat': 0.4, 'fiber': 2.1, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 1.3, 'saturated_fat': 0, 'sodium': 0, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '93902759513197', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'fruit entier (120g)', 'value': 1}, {'id': '94452515327085', 'index': 1, 'nutrition_multiplier': 0.00833333, 'unit': 'gram', 'value': 1}], 'type': 'food', 'user_id': '160406080319149', 'verified': False, 'version': '198055450101605'} []  food
2   {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': ' Banane ()', 'id': 1659839707, 'nutritional_contents': {'calcium': 0.625, 'carbohydrates': 22.84, 'energy': {'unit': 'calories', 'value': 90}, 'fat': 0.33, 'fiber': 2.6, 'iron': 1.857, 'potassium': 358, 'protein': 1.09, 'sodium': 1, 'sugar': 12.23, 'vitamin_a': 0.375, 'vitamin_c': 10.875}, 'public': True, 'serving_sizes': [{'id': '268297681372533', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '268297681372661', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '268847437186549', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '268297673016693', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '163850601983789', 'verified': False, 'version': '129355447387317'}    []  food
3   {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': 'Banane 1 Stück', 'id': 1887842011, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 26.4, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 115}, 'fat': 0.2, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 1.2, 'saturated_fat': 0, 'sodium': 0, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '27521653231597', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 120}, {'id': '28071409045485', 'index': 1, 'nutrition_multiplier': 0.00833333, 'unit': 'g', 'value': 1}, {'id': '27521661620077', 'index': 2, 'nutrition_multiplier': 0.236246, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '234889390534445', 'verified': False, 'version': '53489009870261'}   []  food
4   {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': 'Banane', 'id': 227750309, 'nutritional_contents': {'calcium': 5, 'carbohydrates': 22.8, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.3, 'fiber': 2.6, 'iron': 0.3, 'monounsaturated_fat': 0.1, 'polyunsaturated_fat': 0.1, 'potassium': 358, 'protein': 1.1, 'saturated_fat': 0.1, 'sodium': 1, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 64, 'vitamin_c': 9}, 'public': True, 'serving_sizes': [{'id': '88267487686061', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '88817243499949', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '88267496074541', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '134026256871405', 'verified': True, 'version': '230354056064301'} []  food
5   {'brand_name': 'Banane Ohne Schale', 'country_code': 'DE', 'deleted': False, 'description': 'Banane', 'id': 1889101676, 'nutritional_contents': {'calcium': 0.625, 'carbohydrates': 22.84, 'energy': {'unit': 'calories', 'value': 95}, 'fat': 0.33, 'fiber': 2.6, 'iron': 35.71, 'potassium': 358, 'protein': 1.09, 'sugar': 12.23, 'vitamin_a': 0.375, 'vitamin_c': 10.875}, 'public': True, 'serving_sizes': [{'id': '138151424970349', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '137601669156589', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '138151424970477', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '278614430748141', 'verified': False, 'version': '53074667210277'}    []  food
6   {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Banane Gebacken', 'id': 1349524295, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 25, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 157}, 'fat': 4, 'fiber': 0, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 4, 'saturated_fat': 0, 'sodium': 1, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '59106944525429', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '58557188711669', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '59106944525557', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '58557197100149', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '133324127170493', 'verified': False, 'version': '31634077001709'}    []  food
7   {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Demi banane', 'id': 139561661, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 14, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 93}, 'fat': 2, 'fiber': 0, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 3, 'saturated_fat': 0, 'sodium': 0, 'sugar': 0, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '124397058706493', 'index': 1, 'nutrition_multiplier': 1, 'unit': 'yaourt', 'value': 1}], 'type': 'food', 'user_id': '133476501057517', 'verified': False, 'version': '63530949537133'}    []  food
8   {'brand_name': 'Banane', 'country_code': 'CA', 'deleted': False, 'description': 'Banane (Santé Canada)', 'id': 1568891032, 'nutritional_contents': {'calcium': 1, 'carbohydrates': 27, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 105}, 'fat': 0, 'fiber': 3, 'iron': 2, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 487, 'protein': 1, 'saturated_fat': 0, 'sodium': 1, 'sugar': 14, 'trans_fat': 0, 'vitamin_a': 2, 'vitamin_c': 17}, 'public': True, 'serving_sizes': [{'id': '27092291822629', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'banana 118g', 'value': 1}, {'id': '26542536008869', 'index': 1, 'nutrition_multiplier': 1, 'unit': 'med bananna', 'value': 1}], 'type': 'food', 'user_id': '199722294733869', 'verified': False, 'version': '264028726224173'}   []  food
9   {'brand_name': 'Banane', 'country_code': 'CA', 'deleted': False, 'description': ' Une banane moyenne', 'id': 1484522768, 'nutritional_contents': {'calcium': 1.53, 'carbohydrates': 22.8, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.33, 'fiber': 2, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0.07, 'potassium': 0, 'protein': 1.1, 'saturated_fat': 0.11, 'sodium': 8, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '63099251926181', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '63649007740069', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '63099260314661', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '63649016128549', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '128659968929645', 'verified': False, 'version': '136640012748413'}    []  food

You can drill down further into that json object, (normalize it, etc) to get data in different shapes and forms.