I'm want to access some data from:
https://www.calorie-charts.info/food/all/banana
I tried a Python Request Session but it takes up to 2 minutes to get a response.
import requests
s = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
url = 'https://www.calorie-charts.info/food/all/banana"'
s.headers.update(headers)
r = s.get(url)
print(r.text)
I also tried to find an api link for a XHR Request but I couldn't find one in the Dev Tool - Network Tab.
How can I speed up the process or find a link for a XHR Request?
CodePudding user response:
This is one way to get that data (UPDATED: all 34 pages):
import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
from tqdm import tqdm
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
s = requests.Session()
s.headers.update(headers)
big_df = pd.DataFrame()
for x in tqdm(range(1,35)):
url = f'https://www.calorie-charts.info/food/all/banana/page/{x}'
r = s.get(url)
df = pd.read_html(r.text)[0]
big_df = pd.concat([big_df, df], axis=0, ignore_index=True)
print(big_df)
Result:
Name Calories(kcal) Amount Energy (kj) Proteins (g) Carbohydrates (g) Fat (g) Fiber (g) Unnamed: 8
0 banana 84.5 medium banana (90 g) 354 1 20.0 22.0 2.0 Detail
1 banana bio 105.0 126 g 440 1 27.0 38.0 3.0 Detail
2 banana bar 147.6 40 g 618 6 28.0 4.0 NaN Detail
3 banana curd 224.7 100 g 941 18 30.0 3.0 1.0 Detail
4 ripe banana 199.6 200 g 836 2 40.0 1.0 4.0 Detail
... ... ... ... ... ... ... ... ... ...
1675 Oatmeal lumps of cheese 3 bananas, 1 cheese, 2... 20.5 piece (10 g) 86 78.0 3.0 73.0 36.0 Detail
1676 pancakes (tangerine, apple, banana, 1/2 curd, ... 54.9 1 pancake (71 g) 230 5.0 6.0 1.0 99.0 Detail
1677 Fit cake with nuts, avocados and bananas, almo... 264.4 100 g 1 107 7.0 30.0 13.0 6.0 Detail
1678 100% Whey Protein vanilla, banana, strawberry,... 113.4 portion (30 g) 475 23.0 2.0 1.0 42.0 Detail
1679 Excellent 24% Protein Bar pineapple with cocon... 372.8 85 g 1 561 20.0 36.0 16.0 2.0 Detail
Requests documentation: https://requests.readthedocs.io/en/latest/
Also pandas relevant documentation: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html