I create a crawler to scrape binance.com enter link description here using python selenium. The problem is I want to scrape all the crypto names and their price from this website, But I can't. I am able to scrape only the data which is show on a particular page. Is there any way to grab all the data.

Code trials:

from lib2to3.pgen2 import driver
import schedule
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

def getData():
    driver = webdriver.Chrome(ChromeDriverManager().install())
    driver.maximize_window()
    driver.get("https://www.binance.com/en/markets")
    # driver.implicitly_wait(5)
    bitcoins = driver.find_elements(By.XPATH, "//div[contains(@class, 'css-1ap5wc6')]")
    # prMarCap = driver.find_elements(By.XPATH, "//div[@class='css-leyy1t'] //div[contains(text(), '$')]")
    pr = driver.find_elements(By.XPATH, "//div[@class='css-leyy1t'] //div[@class='css-ydcgk2']")
    MarCap = driver.find_elements(By.XPATH, "//div[@class='css-s779xv']")
    mycoin=[]
    myprice=[]
    mymarcap = []


    for bit in bitcoins:
        # print(bit.text)
        mycoin.append(bit.text)

    for MC in MarCap:
        # print(MC.text)
        mymarcap.append(MC.text)


    for price in pr:
        # print(price.text)
        myprice.append(price.text)
                

    final = zip(mycoin, myprice, mymarcap)

    for data in list(final):
        print(data)

schedule.every(1).seconds.do(getData)
  
while True:
    schedule.run_pending()
    time.sleep(1)

CodePudding user response：

There is no need to scrape it with selenium as you can get it by calling binance api

Here is a code, that saves all <CRYPTO>-BUSD prices to out.json

import json
from collections import defaultdict

import requests

data = requests.get("https://www.binance.com/bapi/asset/v2/public/asset-service/product/get-products?includeEtf=true").json()['data']


crypto = defaultdict(dict)
for value in data:
    if value['s'].endswith("BUSD"):
        crypto[value['b']] = {"HIGH": value['h'], "LOW": value['l'], "CURRENT": value['c']}

with open("out.json", "w") as f:
    f.write(json.dumps(crypto))

CodePudding user response：

To extract and print the Crypto name and their price from binance website you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

driver.get("https://www.binance.com/en/markets")
texts = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@id='tabContainer']//div[@direction='ltr']//div[@data-area='left']//div[@data-bn-type='text']")))]
coins = [texts[i] for i in range(len(texts)) if i % 3 == 1]
prices = [texts[i] for i in range(len(texts)) if i % 3 == 2]
print (coins)
print (prices)

Console Output:

['Bitcoin', 'Ethereum', 'TetherUS', 'BNB', 'USD Coin', 'Ripple', 'Cardano', 'Solana', 'Terra', 'Avalanche', 'BUSD', 'Polkadot', 'Dogecoin', 'SHIBA INU', 'TerraUSD', 'Polygon', 'Wrapped Bitcoin', 'Cosmos', 'Litecoin', 'Uniswap']
['$37,988.60000', '$2,612.69000', '$1.00000', '$363.99000', '$0.99970', '$0.71888', '$0.87973', '$89.56000', '$75.21000', '$74.41000', '$0.99970', '$16.83000', '$0.12316', '$0.00002', '$1.00000', '$1.45000', '$37,989.82000', '$27.26000', '$102.87000', '$9.54000']

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

References

You can find a couple of relevant detailed discussions in:

• Find element by Xpath. How to split the element I don't want inside the Xpath