Web scraping for the website not working using python-CodePudding

My code just keeps running without any results

import requests
import  pandas as pd

url = 'http://www.cmegroup.com/markets/agriculture/livestock/pork-cutout.quotes.html'
Data = requests.get(url)

CodePudding user response：

Seems like an issue with that site specifically requiring header information. I found a solution here that worked for me:

requests.get in python giving connection timeout error

import requests
import  pandas as pd

headers = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}
url = 'http://www.cmegroup.com/markets/agriculture/livestock/pork-cutout.quotes.html'
Data = requests.get(url, timeout=15, verify=False, allow_redirects=True,headers=headers)
print(Data.content)

CodePudding user response：

In this case, your program basically requests the page and then stores the requested data inside the Data variable. You do not do any processing on the variable after. In order to do something with it, you can do something like

print(Data)

This will show you what is inside the variable. You can also use a debugging tool such as those within vscode if you add a breakpoint there.