Solve TypeError: 'NoneType' object is not callable in webscraper-CodePudding

So I'm trying to create a Python script that goes to a website. Looks at all the results of coins on the page. Clicks on all the results and checks if there is a telegram link on the page, if so copy it and add it to the print result, if not just print the coin name. I'm getting a TypeError: 'NoneType' object is not callable error on line 23, in element.click() . I'm not sure how to fix it.

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

# Set up a webdriver to simulate a browser
driver = webdriver.Chrome()

# Send a request to the webpage
response = requests.get('https://www.coingecko.com/en/new-cryptocurrencies')

# Parse the response
soup = BeautifulSoup(response.text, 'html.parser')

# Find all elements with the class "coin-name"
coin_elements = soup.find_all(class_='coin-name')

# Iterate through the coin elements
for element in coin_elements:
  # Extract the text from the element (the coin name)
  coin_name = element.text
  
  # Simulate a click on the element
  element.click()
  
  # Wait for the page to load
  time.sleep(2)
  
  # Parse the content of the coin's page
  coin_soup = BeautifulSoup(driver.page_source, 'html.parser')
  
  # Find the element containing the Telegram group link
  telegram_element = coin_soup.find('a', href=re.compile(r'^https://t\.me/'))
  
  # Check if the element was found
  if telegram_element is not None:
    # Extract the link from the element
    telegram_link = telegram_element['href']
    
    # Print the coin name and link
    print(f'{coin_name}: {telegram_link}')
  else:
    # Print a message if the element was not found
    print(f'{coin_name}: No Telegram group found')

# Close the webdriver
driver.close()

error:

line 23, in <module>
    element.click()
TypeError: 'NoneType' object is not callable

CodePudding user response：

You cannot mix up selenium and BeautifulSoup. This coin_elements = soup.find_all(class_='coin-name') won't give you list of WebElements so you cannot perform click() on it.

See this thread to check what type find_all in BS returns: What is the return type of the find_all method in Beautiful Soup?

CodePudding user response：

use find_elements_by_xpath method for selecting clickable elements. Selenium doesn't accept other datatypes for click action.

You can try something like,

elements = driver.find_elements_by_xpath('//*[@class='coin-name']')
for element in elements:
    element.click()

Note: From your traceback it looks like you have got NoneType, which could be the indicator for inspecting the selectors once.

CodePudding user response：

You don't need to use selenium at all. Using python requests and bs4 you can achieved that.

First fetch all the urls for the coins and then iterate it and get response from the page and get the telegram link.

Code:

import requests
from bs4 import BeautifulSoup
import requests
import re
response = requests.get('https://www.coingecko.com/en/new-cryptocurrencies')
# Parse the response
soup = BeautifulSoup(response.text, 'html.parser')
coin_elements =["https://www.coingecko.com"   item['href'] for item in soup.select("table.sort a[href*='en/coins']")]
#print(coin_elements)
coin_elements_text =[item.text.strip() for item in soup.select("table.sort a[href*='en/coins']")]

for pageurl,coin_name in zip(coin_elements,coin_elements_text):
    r=requests.get(pageurl)
    coin_soup=BeautifulSoup(r.text, 'html.parser')
    telegram_element = coin_soup.select_one("a[rel='nofollow noopener'][href*='t.me']")
    try:
       # Extract the link from the element
       telegram_link = telegram_element['href']    
       # Print the coin name and link
       print(f'{coin_name}: {telegram_link}')
    except:
       # Print a message if the element was not found
       print(f'{coin_name}: No Telegram group found')

Console output:

FinToken


FTC: https://t.me/FintokenEN
Canadian Inuit Dog


CADINU: https://t.me/cadinu
DEXO


DEXO: https://t.me/DexoOfficial
DexWallet


DWT: https://t.me/Dexwalletofficialcommunity
Saracens Fan Token


SARRIES: https://t.me/ChilizTrade
Tsuki no usagi


GYOKUTO: https://t.me/GyokutoEntry
Falcon


FLN: https://t.me/SpaceDexPortal
Dejitaru Shirudo


SHIELD: https://t.me/dejitarushirudoentry
Cronos ID


CROID: No Telegram group found
Chikn Worm


WORM: No Telegram group found
Huckleberry Inu


HKBY: https://t.me/huckleberryinuofficial
Wrapped OAS


WOAS: No Telegram group found
Dogu Inu


DOGU: https://t.me/Dogu_Inu
Harlequins Fan Token


QUINS: https://t.me/ChilizTrade
Sonic Suite


SONIC: https://t.me/sonicsuiteeth
Matchcup


MATCH: https://t.me/Matchcupofficial
Mu Inu


MUINU: No Telegram group found
IMPT


IMPT: https://t.me/IMPTprogram
Radioreum


THERADIO: https://t.me/radioreum
Centurion Invest


CIX: https://t.me/centurioninvestgroup
Liquid ASTR


NASTR: https://t.me/Algem_io
Rabbit2023
...so on