I am trying to parse/scrape https://etherscan.io/tokens website using requests in Python but I get the following error:
etherscan.io
Checking if the site connection is secure
etherscan.io needs to review the security of your connection before proceeding. Ray ID: 73b56fc71bc276ed Performance & security by Cloudflare
Now, I found a solution here: https://stackoverflow.com/a/62687390/4190159 but when I try to use this solution, I am still not being able to read the actual content of the website and getting a different error stated below.
My code as follows:
import requests
from collections import OrderedDict
from requests import Session
import socket
answers = socket.getaddrinfo('etherscan.io', 443)
(family, type, proto, canonname, (address, port)) = answers[0]
s = Session()
headers = OrderedDict({
'Accept-Encoding': 'gzip, deflate, br',
'Host': "grimaldis.myguestaccount.com",
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0'
})
s.headers = headers
response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text
print(response)
Error for the above code as follows:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect ssl_context=context, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl.py", line 390, in ssl_wrap_socket return context.wrap_socket(sock) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in init self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "label_scrapper.py", line 16, in response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 543, in get return self.request('GET', url, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),)) Somdips-MacBook-Pro:Downloads somdipdey$ python3 label_scrapper.py Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect ssl_context=context, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl.py", line 390, in ssl_wrap_socket return context.wrap_socket(sock) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in init self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "label_scrapper.py", line 16, in response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 543, in get return self.request('GET', url, **kwargs) Somdips-MacBook-Pro:Downloads somdipdey$ python3 label_scrapper.py Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect ssl_context=context, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl.py", line 390, in ssl_wrap_socket return context.wrap_socket(sock) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in init self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "label_scrapper.py", line 16, in response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 543, in get return self.request('GET', url, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))
How to resolve this?
CodePudding user response：
The website is under cloudflare protection. So you can use cloudscraper instead of requests to get rid of the protection. Now it's working fine.
Example:
from bs4 import BeautifulSoup
import cloudscraper
scraper = cloudscraper.create_scraper(delay=10, browser={'custom': 'ScraperBot/1.0',})
url = 'https://etherscan.io/tokens'
req = scraper.get(url)
soup = BeautifulSoup(req.content,'lxml')
for tr in soup.select('table#tblResult tbody tr'):
d= list(tr.stripped_strings)
print(d)
Output:
['1', 'Tether USD (USDT)', 'Tether gives you the joint benefits of open blockchain technology and traditional currency by converting your cash into a stable digital currency equivalent.', '$1.005', '0.000042\xa0Btc', '0.000534\xa0Eth', '0.40%', '$56,223,000,912.00', '$67,594,315,221.00', '$40,022,235,444.23', '4,384,118', '0.003%']
['2', 'USD Coin (USDC)', 'USDC is a fully collateralized US Dollar stablecoin developed by CENTRE, the open source project with Circle being the first of several forthcoming issuers.', '$1.006', '0.000042\xa0Btc', '0.000534\xa0Eth', '0.44%', '$7,028,538,289.00', '$53,707,876,130.00', '$46,882,045,425.04', '1,470,173', '0.132%']
['3', 'BNB (BNB)', 'Binance aims to build a world-class crypto exchange, powering the future\nof crypto finance.', '$316.6558', '0.013290\xa0Btc', '0.168135\xa0Eth', '-0.87%', '$1,131,380,099.00', '$51,088,380,246.00', '$5,250,000,297.97', '322,395', '0.001%']
['4', 'Binance USD (BUSD)', 'Binance USD (BUSD) is a dollar-backed stablecoin issued and custodied by Paxos Trust Company, and regulated by the New York State Department of Financial Services. BUSD is available directly for sale 1:1 with USD on Paxos.com and will be listed for trading on Binance.', '$0.9976', '0.000042\xa0Btc', '0.000530\xa0Eth', '-0.30%', '$6,219,951,485.00', '$17,920,238,921.00', '$17,532,299,450.15', '125,632', '0.197%']
['5', 'HEX (HEX)', "HEX.com averages 25% APY interest recently. HEX virtually lends value from stakers to non-stakers as staking reduces supply. The launch ends Nov. 19th, 2020 when HEX
stakers get credited ~200B HEX. HEX's total supply is now ~350B. Audited 3 times, 2 security, and 1 economics.", '$0.0626', '0.000003\xa0Btc', '0.000033\xa0Eth', '-5.10%', '$22,598,794.00', '$10,856,229,132.00', '$36,158,058,204.94', '308,035', '-0.040%']
['6', 'SHIBA INU (SHIB)', 'SHIBA INU is a 100% decentralized community experiment with it claims that 1/2 the tokens have been sent to Vitalik and the other half were locked to a Uniswap pool and the keys burned.', '$0.00', '0.000000\xa0Btc', '0.000000\xa0Eth', '-9.20%', '$2,070,477,368.00', '$9,155,756,506.00', '$15,489,873,503.95', '1,206,115', '0.088%']
['7', 'stETH (stETH)', 'stETH is a token that represents staked ether in Lido, combining the
value of initial deposit staking rewards. stETH tokens are pegged 1:1 to the ETH staked with Lido and can be used as one would use ether, allowing users to earn Eth2 staking rewards whilst benefiting from Defi yields.', '$1,844.28', '0.077404\xa0Btc', '0.979260\xa0Eth', '-2.26%', '$3,408,944.00', '$7,909,446,933.00', '$3,418,574,006.52', '94,316', '0.215%']
['8', 'Matic Token (MATIC)', 'Matic Network brings massive scale to Ethereum using an adapted version of Plasma with PoS based side chains. Polygon is a well-structured, easy-to-use platform for Ethereum scaling and infrastructure development.', '$0.9397', '0.000039\xa0Btc', '0.000499\xa0Eth', '-6.39%', '$529,032,596.00', '$7,551,024,649.00', '$9,397,310,544.00', '466,641', '0.083%']
['9', 'Dai Stablecoin (DAI)', 'Multi-Collateral Dai, brings a lot of new and exciting features, such as support for new CDP collateral types and Dai Savings Rate.', '$1.005', '0.000042\xa0Btc', '0.000534\xa0Eth', '0.36%', '$635,956,564.00', '$6,800,555,162.00', '$9,848,650,590.65', '479,078', '-0.008%']
['10', 'Wrapped BTC (WBTC)', 'Wrapped Bitcoin (WBTC) is an ERC20 token backed 1:1 with Bitcoin.\nCompletely transparent. 100% verifiable. Community led.', '$23,983.00', '1.006567\xa0Btc', '12.734291\xa0Eth', '-1.26%', '$263,338,154.00', '$5,928,934,190.00', '$6,279,085,162.00',
'51,459', '0.058%']
... so on