Scraping opensea with selenium, error: selenium.webdriver.remote.webelement.webelement-CodePudding

CODE

website = 'https://opensea.io/collection/azuki?search[sortAscending]=false&search[sortBy]=FAVORITE_COUNT'

s= Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=s)
driver.get(website)
driver.maximize_window()
time.sleep(1)
see_more = driver.find_element(By.XPATH, '//div[@]').click()

coll_name = driver.find_elements(By.XPATH, '//h1')
coll_desc1 = driver.find_elements(By.XPATH, '(//p[1])[1]')
coll_desc2 = driver.find_elements(By.XPATH, '//p[2]')
coll_desc3 = driver.find_elements(By.XPATH, '//p[3]')
art_name = driver.find_elements(By.XPATH, '//div[@]')
art_price = driver.find_elements(By.XPATH, '//div[@]')

collection_name = []
collection_desc1 = []
collection_desc2 = []
collection_desc3 = []
name = []
price = []

for c in coll_name:
    collection_name.append(c.text)
    time.sleep(1)


for d1 in coll_desc1:
    collection_desc1.append(d1.text)
    time.sleep(1)

for d2 in coll_desc2:
    collection_desc2.append(d2.text)
    time.sleep(1)
    
for d3 in coll_desc3:
    collection_desc3.append(d3.text)
    time.sleep(1)

for n in art_name:
    name.append(n.text)
    time.sleep(1)

for p in art_price:
    price.append(p.text)
    time.sleep(1)

Collection_Azuki = {'Collection_Name': coll_name, 'Collection_Description1': coll_desc1, 'Collection_Des2': coll_desc2, 'Colltionec_Des3': coll_desc3, 'Art_Name_fav': art_name, 'Art_Price_fav': art_price}

df = pd.DataFrame.from_dict(Collection_Azuki, orient='index')
df = df.transpose()
df

1stly error : selenium.webdriver.remote.webelement.webelement

2ndly, I want to get more than 6 results

facing this error after program running

XPATH Working well

errorselenium.webdriver.remote.webelement.webelement

please help me solving these tow issues 1stly and 2ndly

CodePudding user response：

You use wrong variables when you create Collection_Azuki

You use

'Collection_Name': coll_name

which means

'Collection_Name': driver.find_elements(By.XPATH, '//h1')

but you should use

'Collection_Name': collection_name

which uses list created with c.text

The same with other variables in Collection_Azuki

You need

Collection_Azuki = {
    'Collection_Name': collection_name,
    'Collection_Description1': collection_desc1,
    'Collection_Des2': collection_desc2,
    'Colltionec_Des3': collection_desc3,
    'Art_Name_fav': name,
    'Art_Price_fav': price
}

Result:

   Collection_Name                                                          Collection_Description1 Collection_Des2 Colltionec_Des3 Art_Name_fav Art_Price_fav
0            Azuki  Take the red bean to join the garden. View the collection at azuki.com/gallery.                                  Azuki #8740            25
1             None                                                                             None            None            None  Azuki #1948          2,61
2             None                                                                             None            None            None  Azuki #4886            19
3             None                                                                             None            None            None  Azuki #1889            12
4             None                                                                             None            None            None  Azuki #8360            50
5             None                                                                             None            None            None  Azuki #2362          2,94
6             None                                                                             None            None            None  Azuki #4752        18,888
7             None                                                                             None            None            None  Azuki #9024          4,07
8             None                                                                             None            None            None  Azuki #5397         11,75
9             None                                                                             None            None            None  Azuki #4385            15
10            None                                                                             None            None            None  Azuki #5412           5,5
11            None                                                                             None            None            None  Azuki #3222          3,18
12            None                                                                             None            None            None         None            11