NoSuchElementException even when using find_element(By.XPATH, value='xpath

Target: To get titles from https://www.boxofficemojo.com/chart/top_lifetime_gross/?ref_=bo_cso_ac and convert them into a list.

The code:

driver = webdriver.Chrome()
driver.get('https://www.boxofficemojo.com/chart/top_lifetime_gross/?ref_=bo_cso_ac')

xpath = '//td[@]/a[@]'
movie_names = driver.find_element(By.XPATH, value=xpath)

The error:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//td[@]/a[@]"}

Appreciate the effort and help in advance.

CodePudding user response：

You can use a css attribute = value selector as follows. It is a quick method for matching and likely more stable than a long xpath. This targets the title hrefs which contains title.

from selenium import webdriver

d = webdriver.Chrome()
d.get('https://www.boxofficemojo.com/chart/top_lifetime_gross/?ref_=bo_cso_ac')
titles = [i.text for i in d.find_elements_by_css_selector('[href*=title]')]
print(titles)

Given the deprecation warning, here is the by version

from selenium import webdriver
from selenium.webdriver.common.by import By

d = webdriver.Chrome()
d.get('https://www.boxofficemojo.com/chart/top_lifetime_gross/?ref_=bo_cso_ac')
titles = [i.text for i in d.find_elements(by=By.CSS_SELECTOR, value='[href*=title]')]
print(titles)

CodePudding user response：

The titles are the innerText of <a> tags with class a-link-normal and href value starting with /title.

Solution

To extract the titles you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use the following Locator Strategy:

Using CSS_SELECTOR:

driver.get("https://www.boxofficemojo.com/chart/top_lifetime_gross/?ref_=bo_cso_ac")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.a-link-normal[href^='/title']")))])

Console Output:

['Star Wars: Episode VII - The Force Awakens', 'Avengers: Endgame', 'Spider-Man: No Way Home', 'Avatar', 'Black Panther', 'Avengers: Infinity War', 'Titanic', 'Jurassic World', 'The Avengers', 'Star Wars: Episode VIII - The Last Jedi', 'Incredibles 2', 'The Lion King', 'The Dark Knight', 'Rogue One: A Star Wars Story', 'Star Wars: Episode IX - The Rise of Skywalker', 'Beauty and the Beast', 'Finding Dory', 'Frozen II', 'Star Wars: Episode I - The Phantom Menace', 'Star Wars', 'Avengers: Age of Ultron', 'The Dark Knight Rises', 'Shrek 2', 'E.T. the Extra-Terrestrial', 'Toy Story 4', 'Captain Marvel', 'The Hunger Games: Catching Fire', "Pirates of the Caribbean: Dead Man's Chest", 'The Lion King', 'Jurassic World: Fallen Kingdom', 'Toy Story 3', 'Wonder Woman', 'Iron Man 3', 'Captain America: Civil War', 'The Hunger Games', 'Spider-Man', 'Jumanji: Welcome to the Jungle', 'Jurassic Park', 'Transformers: Revenge of the Fallen', 'Frozen', 'Spider-Man: Far from Home', 'Guardians of the Galaxy Vol. 2', 'Harry Potter and the Deathly Hallows: Part 2', 'Finding Nemo', 'Star Wars: Episode III - Revenge of the Sith', 'The Lord of the Rings: The Return of the King', 'Spider-Man 2', 'The Passion of the Christ', 'The Secret Life of Pets', 'Despicable Me 2', 'The Batman', 'The Jungle Book', 'Deadpool', 'Inside Out', 'Aladdin', 'Furious 7', 'Transformers: Dark of the Moon', 'American Sniper', 'The Lord of the Rings: The Two Towers', 'Zootopia', 'The Hunger Games: Mockingjay - Part 1', 'Spider-Man 3', 'Minions', 'Joker', 'Aquaman', 'Spider-Man: Homecoming', 'Alice in Wonderland', 'Guardians of the Galaxy', 'Forrest Gump', 'Batman v Superman: Dawn of Justice', 'It', 'Suicide Squad', 'Deadpool 2', 'Shrek the Third', 'Jumanji: The Next Level', 'Transformers', 'Iron Man', "Harry Potter and the Sorcerer's Stone", 'Indiana Jones and the Kingdom of the Crystal Skull', 'The Lord of the Rings: The Fellowship of the Ring', 'Thor: Ragnarok', 'Iron Man 2', 'Star Wars: Episode II - Attack of the Clones', "Pirates of the Caribbean: At World's End", 'Star Wars: Episode VI - Return of the Jedi', 'Independence Day', 'Pirates of the Caribbean: The Curse of the Black Pearl', 'Skyfall', 'The Hobbit: An Unexpected Journey', 'Harry Potter and the Half-Blood Prince', 'The Twilight Saga: Eclipse', 'The Twilight Saga: New Moon', 'Harry Potter and the Deathly Hallows: Part 1', 'The Sixth Sense', 'Up', 'Star Wars: Episode V - The Empire Strikes Back', 'Inception', 'Harry Potter and the Order of the Phoenix', 'The Twilight Saga: Breaking Dawn - Part 2', 'The Chronicles of Narnia: The Lion, the Witch and the Wardrobe', 'Man of Steel', 'Monsters, Inc.', 'Harry Potter and the Goblet of Fire', 'Home Alone', 'The Hunger Games: Mockingjay - Part 2', 'The Matrix Reloaded', 'The Twilight Saga: Breaking Dawn - Part 1', 'Meet the Fockers', 'The Hangover', 'Gravity', 'The Grinch', 'Sing', 'Monsters University', 'Shrek', 'Despicable Me 3', 'Harry Potter and the Chamber of Secrets', 'The Amazing Spider-Man', 'The Incredibles', 'Jaws', 'How the Grinch Stole Christmas', 'Captain America: The Winter Soldier', 'The Hobbit: The Desolation of Smaug', 'The Lego Movie', 'Star Trek', 'I Am Legend', 'The Blind Side', 'The Hobbit: The Battle of the Five Armies', 'The Hangover Part II', 'Despicable Me', 'Batman', 'Night at the Museum', 'Men in Black', 'Harry Potter and the Prisoner of Azkaban', 'Moana', 'Indiana Jones and the Raiders of the Lost Ark', 'Toy Story 2', 'Transformers: Age of Extinction', 'Cars', 'Ghostbusters', 'Bruce Almighty', 'Twister', 'My Big Fat Greek Wedding', 'Maleficent', 'Pirates of the Caribbean: On Stranger Tides', 'Shrek Forever After', 'Fast & Furious 6', 'Brave', 'Oz the Great and Powerful', 'Beverly Hills Cop', 'X-Men: The Last Stand', 'War of the Worlds', 'Fantastic Beasts and Where to Find Them', 'X-Men: Days of Future Past', 'Cast Away', 'The Exorcist', 'Doctor Strange', 'The Lost World: Jurassic Park', 'Justice League', 'Star Trek Into Darkness', 'The Martian', 'Signs', 'Hancock', 'The Bourne Ultimatum', 'Logan', 'Rush Hour 2', 'The Fate of the Furious', 'Shang-Chi and the Legend of the Ten Rings', 'WALL·E', 'Toy Story', 'Big Hero 6', 'Mission: Impossible - Fallout', 'National Treasure: Book of Secrets', 'Alvin and the Chipmunks: The Squeakquel', 'Mrs. Doubtfire', 'Beauty and the Beast', 'Ted', 'King Kong', 'Ghost', 'How to Train Your Dragon', 'The Da Vinci Code', 'Aladdin', 'Alvin and the Chipmunks', 'Saving Private Ryan', 'Bohemian Rhapsody', 'Ant-Man and the Wasp', "Madagascar 3: Europe's Most Wanted", 'Kung Fu Panda', 'Mission: Impossible II', 'A Star Is Born', 'X2: X-Men United', 'The Lorax', 'Solo: A Star Wars Story', 'Venom: Let There Be Carnage', 'Venom', 'Austin Powers in Goldmember', 'Back to the Future', 'It Chapter Two', '300', 'Coco', 'Fast Five']

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC