How to avoid freeze while scrape from CME webpage?-CodePudding

I am trying to scrape data from CME but the code seems to freeze at requests.get() function.

import requests
from bs4 import BeautifulSoup

URL = 'https://www.cmegroup.com/markets/interest-rates/us-treasury/2-year-us-treasury-note.settlements.html'
page = requests.get(URL)

CodePudding user response：

Seems that they are checking for user-agent

The User-Agent request header is a characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requesting user agent.

Not a specific one, so just give them your favorite agent:

requests.get(URL, headers={'user-agent':'SALT'}).text

More about user-agent check the docs