I am relatively new to web scraping using Python, and I am having a lot of difficulty pulling the name value out of an HTML table row on CoinMarketCap.com. Their structure is unfamiliar to me. I have tried several methods, both on stack overflow and on other sites, to no avail. Here is a snippet of their html: https://i.stack.imgur.com/eBamV.png This is the code I currently have:

import requests
from bs4 import BeautifulSoup

page = requests.get("https://coinmarketcap.com/rankings/exchanges/").text

soup = BeautifulSoup(page, features="html.parser")

tags = soup.findAll("div", class_="sc-16r8icm-0 sc-1teo54s-1 dNOTPP")

tables = soup.findChildren('tr')

my_table = tables[0]

rows = my_table.findChildren(['td'])

print(rows)

for row in rows:
    cells = row.findChildren('td')
    for cell in cells:
        value = cell.string
        print("the value in this cell is %s" % value)

thanks in advance for any help!

CodePudding user response：

The data you see is embedded within the page in Json form. To parse it you could use next example:

import json
import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://coinmarketcap.com/rankings/exchanges/"

soup = BeautifulSoup(requests.get(url).content, "html.parser")
data = soup.select_one("#__NEXT_DATA__").text
data = json.loads(data)

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

df = pd.json_normalize(data["props"]["initialProps"]["pageProps"]["exchange"])

print(df.head().to_markdown())

Prints:

CodePudding user response：

These sc-16r8icm-0 sc-1teo54s-1 dNOTPP are three classes separated with spaces. If you need to identify an element by multiple classes, use a selector like this

tags = soup.select("div.sc-16r8icm-0.sc-1teo54s-1.dNOTPP")