How to scrape data from ul li tag including links of pages with selenium python?-CodePudding

This is the link where I want to get the data, I want to get the multipage data in this link

I want to get all the data that has been red spaced with 33 pages as above, but I don't know how to click through the page to get the data back on page 2 and many more pages after that.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(r'C:\SeleniumDrivers\chromedriver.exe')
 wait = WebDriverWait(driver, 10)

driver.get("https://scan.multichain.org/#/tokens")



el = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 
"li.primary")))
el_hrefs = el.find_element(By.XPATH,".//descendant::a[@href]")
for i in el_hrefs:
    print(i.get_attribute('href'))

But not working for me

CodePudding user response：

You can modify the sleep time to adjust the speed of retrieval as required. If you are getting errors, you may want to slow it down by increasing the sleep times.

This will add each row to the data list. You will then have to process this data later, once extracted. In the code below, I only extracted 3 pages, however, you can increase the number of pages as required.

Code:

from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By

# Function to extract the data
def get_data(my_driver):
    sleep(5)
    body = my_driver.find_element(By.CLASS_NAME, "custom-default-table-tbody").text
    return (body.split('\n'))


# Open Browser and get webpage
driver = webdriver.Chrome()
driver.get("https://scan.multichain.org/#/tokens")

# Extract data from each page (out of 3 pages)
data = []
pages = 3
for i in range(pages):
    data.extend(get_data(driver))
    
    # Click the right arrow
    sleep(1)
    right_arrow = driver.find_element(By.CSS_SELECTOR, 'div#root div.custom-default-table-title > ul > li:nth-child(4)')
    right_arrow.click()

# Print each record
header = ['recordNum', 'name', 'Token Address', 'Underlying', 'Chain', 'Supply', 'TVL (USD)']
print(header)
for i in data:
    print(i)

Output:

['recordNum', 'name', 'Token Address', 'Underlying', 'Chain', 'Supply', 'TVL (USD)']
1 USDCoin 0x04068da6…335d5b75 --- Fantom 241,221,956.20 USDC 241,463,178.16
2 MAI(miMATIC) 0xfb98b335…62ea213b --- Fantom 114,760,090.57 MAI 114,044,905.69
3 Magic Internet Money 0x13096662…8cb8c18d --- Avalanche 111,935,872.06 MIM 112,047,807.93
4 USDC 0x7ea2be2d…5e329d29 0xa0b86991…3606eb48 Ethereum 105,836,469.08 anyUSDC 105,942,305.55
5 TetherUSD 0x049d6802…633a3c7a --- Fantom 83,551,593.27 USDT 83,635,144.87
6 DaiStablecoin 0x8d11ec38…ef8abf3e --- Fantom 83,545,505.35 DAI 83,545,505.35
7 MagicInternetMoney 0x82f0b8b4…4b6829c1 --- Fantom 65,318,864.15 MIM 65,384,183.01
8 TetherUSD 0x22648c12…a4154ebc 0xdac17f95…3d831ec7 Ethereum 50,084,271.13 anyUSDT 50,134,355.40
9 WrappedMEMO 0xddc03851…c0dfeb52 --- Fantom 1,714.64 wMEMO 48,702,714.05
10 Anyswap 0xf99d58e4…1861b4d6 --- Ethereum 7,573,507.04 ANY 35,368,277.87
11 WrappedBTC 0x321162cd…04051b11 --- Fantom 1,758.74 WBTC 29,624,644.19
12 WrappedEther 0x74b23882…6b58c76d --- Fantom 23,006.98 WETH 28,058,387.24
13 Wemix 0x80a16016…9ff0f844 0x7d72b22a…d6ec1c5f WEMIX3.0 69,999,927.79 anyWEMIX 25,967,593.21
14 DaiStablecoin 0x739ca6d7…fa8abc4b 0x6b175474…95271d0f Ethereum 25,100,649.22 anyDAI 25,100,649.22
15 MagicInternetMoney 0xfea7a6a0…a57d6c2a --- Arbitrum 24,217,952.80 MIM 24,242,170.75
16 Anyswap 0xf68c9df9…bad0b6fa --- BNB CHAIN 4,135,652.81 ANY 19,313,498.60
17 Aurox Token 0xb2c7a198…b2034219 0xc6dddb5b…a807260b BNB CHAIN 901,714.00 anyURUS 19,170,439.64
18 Magic Internet Money 0xb153fb3d…eec57901 --- Optimism 19,149,272.65 MIM 19,168,421.92
19 USDCoin 0xe3f5a90f…a1a0ad7d --- Moonriver 17,632,033.94 USDC 17,649,665.97
20 Multichain 0x9fb9a339…3c8764e3 --- BNB CHAIN 3,508,921.28 MULTI 15,684,878.14
21 (PoS)TetherUSD 0xe3eeda11…e61d3cac 0xc2132d05…04b58e8f Polygon 15,211,223.55 anyUSDT 15,226,434.77
22 USDCoin 0x818ec0a7…e6d6dc0b --- Moonbeam 14,549,419.81 USDC 14,563,969.22
23 Baby Doge Coin 0x97513e97…90b163c5 --- OKC 13,862,840,159,165,384.00 BabyDoge 12,799,338.51
24 Fantom 0x4e15361f…73717870 --- Ethereum 48,675,564.09 FTM 9,939,404.16
25 Dogechain Token 0x7b4328c1…09cf9180 --- Ethereum 9,746,495,166.89 DC 8,954,494.97
26 PolkastarterToken 0x7e624fa0…0887f570 --- BNB CHAIN 24,855,627.87 POLS 8,707,199.86
27 USD Coin 0xe3f5a90f…a1a0ad7d --- WEMIX3.0 7,870,169.18 USDC 7,878,039.35
28 IronBank 0x00a35fd8…95870dfb --- Optimism 5,247,696.61 IB 7,504,206.15
29 Ethereum 0x639a647f…92c62c5c --- Moonriver 6,055.80 ETH 7,385,411.49
30 Marlin POND 0xb53ec4ac…4777cd65 0x57b94600…0e58d26c Ethereum 923,045,080.14 anyPOND 6,685,458.60
31 Mai Stablecoin 0x65e66a61…0d84d97a 0xdfa46478…2e964b02 Optimism 6,551,104.63 anyMAI 6,510,278.15
32 DxSale.Network 0xb969792e…aee1d01d 0x04f73a09…52f0d245 BNB CHAIN 44,451,400.00 anySALE 6,226,352.05
33 Anyswap 0xddcb3ffd…bab34239 --- Fantom 1,315,442.12 ANY 6,143,114.68
34 MediaLicensingToken 0x4518231a…5b583263 --- BNB CHAIN 78,404,095.54 MLT 5,265,775.86
35 USD Coin 0xfa9343c3…29f8a40f --- Kava 5,101,447.91 USDC 5,106,549.35
36 Baby Doge Coin 0x8b92ea35…549b7d6b 0xac57de9c…db0d460b Ethereum 5,171,603,000,000,000.00 anyBabyDoge 4,774,858.30
37 Tether USD 0xb44a9b69…959c663c --- Kava 4,680,423.86 USDT 4,685,104.28
38 Beefy.Finance 0xd6070ae9…6179d960 --- Fantom 14,215.74 BIFI 4,508,380.11
39 Parsiq Token 0xd21d29b3…93d2a577 --- BNB CHAIN 57,536,307.53 PRQ 4,492,262.28
40 SAITO 0x3c6dad04…1be134da --- BNB CHAIN 262,857,024.86 SAITO 4,293,114.99
41 UniDex 0x2130d2a1…43ba36e0 --- Fantom 1,162,309.47 UNIDX 4,126,198.62
42 beefy.finance 0xfbdd1943…165d01b8 --- Polygon 12,819.11 BIFI 4,065,451.70
43 USD Coin 0xe3f5a90f…a1a0ad7d --- Step Network 3,952,415.09 USDC 3,956,367.51
44 Dai Stablecoin 0x765277ee…99b39c61 --- Kava 3,836,172.22 DAI 3,836,172.22
45 Multichain 0x9fb9a339…3c8764e3 --- Fantom 840,781.85 MULTI 3,758,294.89
46 Voucher Ethereum 0x3b25bc1d…561d4764 --- Moonriver 3,508.18 vETH 3,515,126.70
47 TetherUSD 0xb44a9b69…959c663c --- Moonriver 3,458,483.45 USDT 3,461,941.93
48 Wrapped BTC 0xe6b9d092…decc5746 0x2260fac5…3bc2c599 Ethereum 201.29 anyWBTC 3,390,526.61
49 ChainLinkToken 0xb3654dc3…574fbdc8 --- Fantom 551,757.71 LINK 3,305,028.67
50 EarnDefiCoin 0x5ec0d875…5a947b9c 0xa7840988…c7637996 BNB CHAIN 1,079,197.00 anyEDC 3,205,215.09
51 LABS Group 0x134b372f…8274bddf --- BNB CHAIN 7,956,360,240.40 LABS 3,203,230.63
52 Betswap.gg 0x63682bdc…c9763f89 --- Avalanche 1,702,425,801.07 BSGG 3,129,773.64
53 Mai Stablecoin 0xc412eccc…78c4114a 0x3f56e0c3…9b3dea0d BNB CHAIN 3,069,420.00 anyMAI 3,050,291.37
54 MAI(miMATIC) 0xaab16888…128a5a77 0xfb2019df…2b00fc2c Moonriver 3,000,000.00 anyMAI 2,981,304.00
55 USDC 0x8965349f…48abe2a2 0x8ac76a51…32cd580d BNB CHAIN 2,941,968.03 anyUSDC 2,944,910.00
56 USD Coin 0xcc9b1f91…165e0cad 0xa7d7079b…c1a4c664 Avalanche 2,470,609.86 anyUSDC 2,473,080.47
57 GeoDBCoin 0x6b26780e…9c1679ea 0x147faf8d…19126750 Ethereum 697,789,829.90 anyGEO 2,431,441.68
58 Travala.com Token 0x442b153f…1abda1d0 --- Ethereum 4,509,971.81 AVA 2,418,301.00
59 MagicInternetMoney 0xfe19f0b5…19f433ba --- BNB CHAIN 2,323,672.92 MIM 2,325,996.59
60 Astrafer 0x97bbbc5d…74106f17 --- Ethereum 1,451,236.41 ASTRAFER 2,278,441.16
61 beefy.finance 0x4e720dd3…c66f4642 --- Optimism 7,013.45 BIFI 2,224,245.97
62 RFOX 0x0a3a2135…6c0457f5 --- BNB CHAIN 389,994,872.76 RFOX 2,158,707.42
63 Aurox Token 0x78261724…4660cee5 0xc6dddb5b…a807260b Ethereum 98,285.90 anyURUS 2,089,558.23
64 ARTH Valuecoin 0x4b641f60…91f38f90 0xe5250918…5c44d590 Polygon 1,000,475.74 anyARTH 2,020,961.00
65 ARTH Valuecoin (BSC) 0x41a4c529…2f920fcd 0x85dab10c…fe9eaa79 BNB CHAIN 1,000,000.00 anyARTH.bsc 2,020,000.00
66 Mai Stablecoin 0xf44fb887…b53ed557 0xdfa46478…2e964b02 Moonbeam 2,000,100.00 anyMAI 1,987,635.38
67 Mai Stablecoin 0xc9baa8cf…dadc2055 0x3f56e0c3…9b3dea0d Gnosis 2,000,003.31 anyMAI 1,987,539.29
68 Mai Stablecoin 0x2c78f1b7…d43aa07e 0xdfa46478…2e964b02 Metis 2,000,000.00 anyMAI 1,987,536.00
69 Mai Stablecoin 0xabd38032…d00be488 0x3f56e0c3…9b3dea0d IoTeX 2,000,000.00 anyMAI 1,987,536.00
70 xcRMRK 0x471ea49d…307506e4 --- Ethereum 1,072,760.64 xcRMRK 1,973,879.57
71 Bit Store Coin 0x65d9033c…1bbe1732 --- BNB CHAIN 83,982,135.99 STORE 1,943,114.84
72 Spell Token 0xce1bffbd…1a2f7814 --- Avalanche 3,552,707,515.46 SPELL 1,931,109.70
73 USDCoin 0x765277ee…99b39c61 --- Dogechain 1,917,008.24 USDC 1,918,925.25
74 USD Coin 0x818ec0a7…e6d6dc0b --- Telos 1,891,896.83 USDC 1,893,788.73
75 Space Token 0x375488f0…cde736bf --- IoTeX 82,298,608.68 SPACE 1,868,531.48
76 SuperFarm 0x51ba0b04…cc4f0d4d --- BNB CHAIN 20,762,292.84 SUPER 1,750,115.95
77 Alchemix USD 0xe5130d3d…4d8e4c27 --- Fantom 1,705,239.71 alUSD 1,679,320.06
78 Betero 0x7faaf8d4…58349dbb --- BNB CHAIN 808,563,353.14 BTE 1,677,405.10
79 XEND 0x4a080377…1b500608 --- BNB CHAIN 79,947,861.72 XEND 1,645,040.78
80 Binance 0xd67de0e0…c5850454 --- Fantom 6,689.98 BNB 1,625,731.26
81 beefy.finance 0x99c409e5…b8f0baae --- Arbitrum 5,045.97 BIFI 1,600,280.02
82 Bolide 0x766afcf8…a6d84617 --- BNB CHAIN 76,179,166.11 BLID 1,596,935.48
83 Cryptonovae 0x7e78d327…dbccd8e0 0x4ee438be…c5e71eaf Polygon 100,000,000.00 anyYAE 1,584,265.00
84 Klaytn 0x461d5276…30c94eb5 --- WEMIX3.0 9,463,068.31 KLAY 1,554,299.51
85 IceToken 0xf16e81dc…55b87df9 --- Fantom 3,572,904.36 ICE 1,547,985.83
86 Ethereum 0xbdc8fd43…2a532303 0x74b23882…6b58c76d Fantom 1,252.15 anyETH 1,527,077.20
87 OpenLeverage 0xa865197a…654341f3 --- BNB CHAIN 125,828,258.93 OLE 1,507,851.62
88 Binance-Peg BUSD 0x5d9ab552…56167818 --- Moonriver 1,495,381.73 BUSD 1,496,877.11
89 Mai Stablecoin 0xc9baa8cf…dadc2055 0xdfa46478…2e964b02 Aurora 1,500,059.91 MAI 1,490,711.54
90 DEUS 0xf7b38053…e04455f8 0xde5ed76e…65109e44 Fantom 18,000.00 anyDEUS 1,477,440.00
91 UniDex 0x5d47baba…cc01023c --- Optimism 411,072.54 UNIDX 1,459,307.53
92 beefy.finance 0xd6070ae9…6179d960 --- Avalanche 4,573.12 BIFI 1,450,319.78
93 Tether USD 0xfa9343c3…29f8a40f --- Step Network 1,422,393.99 USDT 1,423,816.39
94 Qi Dao 0xe705af5f…35d12890 0x580a84c7…0c241ff4 Polygon 15,620,171.32 anyQI 1,385,274.89
95 STEP.APP 0x332730a4…3cfa41ff 0xb58a9d59…6686d1b6 Step Network 50,900,063.00 anyFITFI 1,372,308.45
96 Frax Share 0x685b63cf…49aa192d 0x3432b6a6…d9c964d0 Ethereum 285,075.00 anyFXS 1,359,807.76
97 GoCryptoMe 0xf545cd9e…6f00af0a 0x9528cceb…baf58a30 Ethereum 88,357,345.54 anyGCME 1,359,474.07
98 Cryptonovae 0x26519efc…7e0630a6 0x4ee438be…c5e71eaf BNB CHAIN 82,621,561.44 anyYAE 1,308,944.48
99 Alchemy 0xbc7d6b50…7003056d --- BNB CHAIN 155,291,670.92 ACH 1,300,873.67
100 AlpacaToken 0xad996a45…445a4302 --- Fantom 5,076,219.66 ALPACA 1,290,573.01
101 WootradeNetwork 0x6626c47c…6d5e8d8a --- Fantom 10,928,634.26 WOO 1,251,678.34
102 Binance 0x264c1383…e204a493 --- Avalanche 5,109.62 BNB 1,241,688.41
103 yearn.finance 0x29b0da86…8ac0ec69 --- Fantom 224.01 YFI 1,234,085.19
104 Curve DAO Token 0x1e4f97b9…9019c68b --- Fantom 2,276,147.84 CRV 1,186,382.88
105 Wrapped MEMO 0x3b79a282…2f7faf57 --- Ethereum 40.88 wMEMO 1,161,189.91
106 Bitrise 0xefaeee33…cdd40d73 0x0eb9036c…a0e3f710 Bitgert 3,641,344,484,197.03 BRISE 1,148,767.72
107 Binance-Peg BUSD 0x332730a4…3cfa41ff --- OKC 1,122,355.60 BUSD 1,123,477.96
108 Plato Game 0x86b3f23b…f8ffd255 0x14885de0…b40baa73 Huobi 16,005,921,360.00 anyPLATO 1,095,605.32
109 Binance-PegBSC-USD 0xedf0c420…81f5f21a 0x55d39832…b3197955 BNB CHAIN 1,091,492.99 anyUSDT 1,092,584.48
110 MetaShooter 0x4e840aad…c128d9ea 0x8057687c…4aecd322 OKC 94,000,000.00 anyMHUNT 1,072,916.94
111 FLOKI 0xb83003f7…0d971525 0xfb5b838b…5363d37e BNB CHAIN 157,544,676,003.70 anyFLOKI 1,052,398.44
112 Dai Stablecoin 0x80a16016…9ff0f844 --- Moonriver 1,039,139.07 DAI 1,039,139.07
113 Gains 0xd9ea5835…5b07de01 --- BNB CHAIN 12,754,135.65 GAINS 1,019,986.49
114 Ridotto 0xe9c64384…b545e3d5 --- BNB CHAIN 45,827,180.39 RDT 1,001,507.20
115 KCC-Peg USD Coin 0x375488f0…cde736bf 0x980a5afe…ed3c3430 KCC 999,942.10 anyUSDC 1,000,942.04
116 KCC-Peg Tether USD 0xf390830d…fc5dcbc2 0x0039f574…111baf48 KCC 999,099.80 anyUSDT 1,000,098.90
117 TOR 0x80d20922…91f0ef4d 0x1d6cbdc6…52b8ca83 BNB CHAIN 1,000,000.00 anyTOR 995,873.00
118 Mai Stablecoin 0x4b641f60…91f38f90 0x8d6cebd7…3909fad6 Ethereum 1,000,000.08 anyMAI 993,768.08
119 MAI(miMATIC) 0xc931f61b…71d4ab50 0x2ae35c8e…166ef8cb Cronos 1,000,000.01 anyMAI 993,768.01
120 Mai Stablecoin 0x6b7a8789…e7e71522 0xb84df109…d336afc7 Dogechain 1,000,000.00 anyMAI 993,768.00
121 Mai Stablecoin 0xc931f61b…71d4ab50 0xb9c8f0d3…3cd610ec Celo 1,000,000.00 anyMAI 993,768.00
122 HotbitToken 0x4e840aad…c128d9ea --- BNB CHAIN 202,304,658.40 HTB 990,594.88
123 Decentralized USD 0xcf799767…241fa25c 0x0c10bf8f…9160b5c6 Ethereum 1,000,000.00 anyUSDD 985,028.00
124 Decentralized USD 0xcf799767…241fa25c 0xd1747999…d265be9c BNB CHAIN 1,000,000.00 anyUSDD 985,028.00
125 Decentralized USD_TRON 0xcf799767…241fa25c 0x17f235fd…7c37a6d1 BitTorrent 1,000,000.00 anyUSDD_t 985,028.00
126 Tether USD 0xefaeee33…cdd40d73 --- Telos 982,803.17 USDT 983,785.97
127 L3USD 0xfd6aab8d…132977af 0xdea12c8c…56f7f9d1 BNB CHAIN 1,000,000.00 anyL3USD 978,039.00
128 L3USD 0x7f12a37b…ee0c462b 0x2c2d8a07…53a90d5f Polygon 1,000,000.00 anyL3USD 978,039.00
129 L3USD 0x84cebca6…2852b1dc 0x5f0456f7…00724c6a Fantom 1,000,000.00 anyL3USD 978,039.00
130 Request Token 0xb25e20de…e8a94762 --- Polygon 10,392,461.49 REQ 914,754.85
131 MetaShooter 0xc8f88478…9d438b6c 0xf460d98a…a8038aa2 Ethereum 80,000,000.00 anyMHUNT 913,120.80
132 dotmoovs 0x0ebd9537…e7191469 --- BNB CHAIN 330,848,089.55 MOOV 900,270.74
133 TrueUSD 0x9879abde…43556b7a --- Fantom 879,510.48 TUSD 880,389.99
134 Pancake GAMES 0xe77b2e03…825cf81f 0x61d5822d…9480c8f6 BNB CHAIN 1,500,274,431,366.28 anyGCAKE 879,157.82
135 yAxis 0x02a2b736…92fbbb40 0x91a17008…48acbd91 Avalanche 1,500,000.00 anyYAXIS 864,070.50
136 WEMIX 0x2c69095d…31624cea --- Ethereum 2,262,814.37 WEMIX 839,427.19
137 TrustSwap Token 0x82443a77…8a596245 --- BNB CHAIN 4,863,097.25 SWAP 831,677.17
138 MetaShooter 0x70d9b745…84555630 0x2c717059…adce6d8c BNB CHAIN 72,667,500.00 anyMHUNT 829,427.57
139 DeRaceToken 0x373e768f…45c6d25b --- BNB CHAIN 5,729,800.22 DERC 827,090.93
140 MogulStars 0xbd83010e…cf9f6f9a --- BNB CHAIN 320,202,735.67 STARS 812,536.86
141 Coinspaid 0x2406dce4…0f7879a2 --- BNB CHAIN 25,278,538.80 CPD 791,579.49
142 DappRadar 0x489580eb…77e24965 --- BNB CHAIN 108,971,482.15 RADAR 780,334.98
143 Frax Share 0xd67de0e0…c5850454 --- Avalanche 162,759.45 FXS 776,362.57
144 Pendle 0xfb98b335…62ea213b --- Avalanche 16,071,250.48 PENDLE 763,777.34
145 DeRaceToken 0xb35fcbcf…dcdc60e6 --- Polygon 5,176,992.95 DERC 747,293.76
146 Oath Token 0x39fde572…40f51205 --- Optimism 9,029,185.67 OATH 732,736.48
147 BitBase 0x6fefd97f…7542f9a8 --- BNB CHAIN 4,233,991.88 BTBS 728,458.30
148 Kyber Network Crystal v2 0x39fc9e94…89f50f5f --- Avalanche 1,375,376.41 KNC 727,050.10
149 Bondly 0x5d0158a5…a2e87563 --- BNB CHAIN 129,504,913.28 BONDLY 720,152.22
150 Kalmar Token 0x4f851750…fc29a934 --- Fantom 1,241,965.95 KALM 719,889.42

Note: You will not be able to add this directly into a dataframe, because some items within the name column have names separated by spaces. So you will probably have to process using regex or something similar if that is your plan.