Twitter Scraper newbie-CodePudding

Okay, I am new to coding so please bear with me. I appreciate all the help.

I want to create my own Twitter Scraper using Edge as my browser. My fist problem is that some words arent coloured. For example .webdriver.common.keys should be blue like in the video. (I put a link to the video in the file at the bottom I was watching for reference .

2nd Problem I keep getting the error message to upgrade my selenium from 3 to 4 and I am pretty sure I already have the selenium 4 version. So I dont get why I get this error message.

3rd problem how do I use/apply xpath on here to search right element. Do i need to import from library or update anaconda navigator? I am lost.

Appreciate all the help.

Kind regards,

Squaddo.

BELOW IS MY CODE:

import csv
from getpass import getpass
from time import sleep
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from msedge.selenium_tools import Edge, EdgeOptions

options = EdgeOptions()
options.use_chromium = True
driver = Edge(options=options)

C:\Users\Cagri\AppData\Local\Temp\ipykernel_13256\875207683.py:3: DeprecationWarning: Selenium Tools for Microsoft Edge is deprecated. Please upgrade to Selenium 4 which has built-in support for Microsoft Edge (Chromium): https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/#upgrading-from-selenium-3
  driver = Edge(options=options)

driver.get('https://www.twitter.com/login')

username = driver.find_element_by_xpath('//input[@name="text"]')
username.send_keys('DataForCagri')

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_13256\3481032478.py in 
----> 1 username = driver.find_element_by_xpath('//input[@name="text"]')
      2 username.send_keys('DataForCagri')

AttributeError: 'WebDriver' object has no attribute 'find_element_by_xpath'

LINK TO THE YOUTUBE VIDEO!!!!
https://www.youtube.com/watch?v=3KaffTIZ5II&t=250s

I am really lost, dont have any friends nor family that does coding. So here I am requesting help from strangers. I appreciate all the help!

CodePudding user response：

All the methods like find_element_by_name, find_element_by_xpath, find_element_by_id etc. are deprecated now.
You should use find_element(By. instead.
So, instead of

username = driver.find_element_by_xpath('//input[@name="text"]')

it should be now

username = driver.find_element(By.XPATH, '//input[@name="text"]')

Also I do not have to use the username variable. The text can be sent directly to the returned web element as following:

driver.find_element(By.XPATH, '//input[@name="text"]').send_keys('DataForCagri')

CodePudding user response：

'Selenium Tools' is deprecated in Selenium 4, so you have to import the following modules.

You need to import Service also:

from selenium import webdriver
# for EdgeOptions
from selenium.webdriver.edge.options import Options

from selenium.webdriver.edge.service import Service

options = webdriver.EdgeOptions()
driver = webdriver.Edge(service=Service(<edge driver path>), options=options)

For the error - 'AttributeError: 'WebDriver' object has no attribute 'find_element_by_xpath'', you can refer Prophet's answer.