Main.py

import cloudscraper

scraper = cloudscraper.create_scraper()
cs_recent_page = scraper.get('https://coinsniper.net/new').text

Error

cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.

I'm trying to access this site in order to scrape data off of it, but it uses cloudfare protection, so I found this solution but keep getting this error, is there another way to access this site?

CodePudding user response：

You could use ScrapingAnt instead of cloudscraper - it's pretty useful unless you need to scrape a very large number of pages (the free tier only allows 10k requests per month). If you register, they give you a token that you can paste into the code below:

import requests
import urllib.parse
from bs4 import BeautifulSoup

sa_key = 'YOUR_API_TOKEN' # paste here
sa_api = 'https://api.scrapingant.com/v2/general'
qParams = {'url': 'https://coinsniper.net/new', 'x-api-key': sa_key}
reqUrl = f'{sa_api}?{urllib.parse.urlencode(qParams)}'  

r = requests.get(reqUrl)
# print(r.text) # --> html
soup = BeautifulSoup(r.content, 'html.parser')

I printed with my own function print(prettify_htmlText(soup)) (I sometimes find it more readable than just print(soup.text)) to get the output

New Listings
- CoinSniper
Limited Presale
Token Revolutionising the $850b Carbon Credits Industry
Read More
ad
BNB Price: $342.32
Projects Listed: 39,062
Total Votes: 114,961,735
Watchlists: 1,589,556
Total KYCs: 227
Total Audits: 245
Supported Chains: 8
Register / Login
26K
50K
Submit Coin
Account
Watchlist - Login - Register
Cryptocurrencies
Today's best - All time Best - New listings - By Marketcap - Presales
CoinSniper
Submit coin - Update Coin - Contact Us
Services
Advertising - Audit partners - KYC verification - AMA hosting
Follow us on twitter!
Join our telegram group!
Best Coins Today - All Time Best - New Listings - By Marketcap - Presales
Your banner here? Contact us!
Buy now Buy now Buy now       Buy now Buy now Buy now       Buy now
New listings
Today's best - All time best - New listings - By marketcap - Presales
Promoted
Your coin here? Contact us!
| # Coin                            | Votes        | #  | Name                            | Badges    | Chain | Market Cap | Price            | Change 24h | Launch      | Votes   |
| --------------------------------- | ------------ | -- | ------------------------------- | --------- | ----- | ---------- | ---------------- | ---------- | ----------- | ------- |
| 1 BIG Big Eyes                    | 803,087 VOTE | 1  | BIG Big Eyes                    | KYC Audit | ETH   | Presale    | -                | -          | In 3 months | 803,087 |
| 2 D2T Dash 2 Trade                | 756,164 VOTE | 2  | D2T Dash 2 Trade                | KYC Audit | ETH   | Presale    | -                | -          | In 22 days  | 756,164 |
| 3 IMPT IMPT.io                    | 753,262 VOTE | 3  | IMPT IMPT.io                    | KYC Audit | ETH   | Presale    | -                | -          | In 3 months | 753,262 |
| 4 RIA Calvaria: Duels of Eternity | 752,429 VOTE | 4  | RIA Calvaria: Duels of Eternity | KYC Audit | ETH   | Presale    | -                | -          | In 2 months | 752,429 |
| 5 METRO Metropoly                 | 656,951 VOTE | 5  | METRO Metropoly                 | Audit     | ETH   | Presale    | -                | -          | In 1 month  | 656,951 |
| 6 DOGET Dogeliens                 | 408,400 VOTE | 6  | DOGET Dogeliens                 | Audit     | BSC   | Presale    | -                | -          | In 4 months | 408,400 |
| 7 BABYBTC BabyBitcoin             | 335,483 VOTE | 7  | BABYBTC BabyBitcoin             | KYC Audit | BSC   | Presale    | -                | -          | In 1 month  | 335,483 |
| 8 TFT Toon Finance                | 293,482 VOTE | 8  | TFT Toon Finance                |           | ETH   | Presale    | -                | -          | In 1 month  | 293,482 |
| 9 GGT GameGuru                    | 130,041 VOTE | 9  | GGT GameGuru                    | KYC Audit | BSC   | $ 769,182  | $ 0. (0x2) 8096  | 21.8 %     | 1 month ago | 130,041 |
| 10 NITRO NITRO                    | 10,777 VOTE  | 10 | NITRO NITRO                     | KYC Audit | BSC   | Presale    | -                | -          | In 11 days  | 10,777  |
| 11 BERGERDOGE Berger Dog Coin     | 917 VOTE     | 11 | BERGERDOGE Berger Dog Coin      |           | BSC   | -          | $ 0. (0x12) 5663 | 23.9 %     | 3 days ago  | 917     |
Coins
Chain
ALL
ALL - BSC - ETH - MATIC - TRX - FTM - KCC - DOGECHAIN - OTHER
Platform
ALL
ALL - CMC - CG - CMC & CG
KYC
ALL
ALL - YES - NO
Audit
ALL
ALL - YES - NO
Reset Filters
| # Coin                           | Votes 24h | Votes | #  | Name                          | Chain | Market Cap      | Price            | Change 24h | Launch       | Votes | Votes 24h | Submitted      |
| -------------------------------- | --------- | ----- | -- | ----------------------------- | ----- | --------------- | ---------------- | ---------- | ------------ | ----- | --------- | -------------- |
| 1 NOVA NovaChain                 | 0 VOTE    | 0     | 1  | NOVA NovaChain                | BSC   | -               | -                | -          | Today        | 0     | 0         | 51 minutes ago |
| 2 JUNP4EVER Jump forever         | 0 VOTE    | 0     | 2  | JUNP4EVER Jump forever        | MATIC | -               | -                | -          | 1 day ago    | 0     | 0         | 1 hour ago     |
| 3 LITE THE LITE                  | 0 VOTE    | 0     | 3  | LITE THE LITE                 | BSC   | -               | $ 0. (0x5) 2744  | 10.73 %    | 2 days ago   | 0     | 0         | 1 hour ago     |
| 4 FAMUS likefamus                | 2 VOTE    | 2     | 4  | FAMUS likefamus               | BSC   | $ 3,917,503     | $ 0. (0x3) 3917  | 12.97 %    | 7 days ago   | 2     | 2         | 2 hours ago    |
| 5 MCT