So I'm beginner at selenium again....
I have a problem of getting the images in a web page here 
As you see I'm trying to get the style background image but I can't get it with these lines of code that I've search.
containers = driver.find_elements(by="xpath",value='//div[@]')
news_link_init = []
news_intro_init = []
news_full_init = []
news_image_init = []
for container in containers:
news_link = container.find_element(by="xpath", value="./div[@class='wrap']//h3/a").get_attribute("href")
news_intro = container.find_element(by="xpath", value="./div[@class='wrap']//span[@class='intro']").text
news_full = container.find_element(by="xpath", value="./div[@class='wrap']//span[@class='full']").text
// AS YOU SEE HERE I"M TRYING TO GET THE ATTRIBUTE STYLE BUT I CAN"T GET IT AND IT GIVES ME ERROR HERE
news_image = container.find_element(by="xpath", value="./div[@class='thumbnail']").get_attribute("style")
news_link_init.append(news_link)
news_intro_init.append(news_intro)
news_full_init.append(news_full)
// Append images
news_image_init.append(news_image)
As you see here in the code I'm trying to get those background-image:url() but I can't..can anyone help me on these ? Thanks.
Update: This is the always error I have receive based on two answers comment down below
CodePudding user response：
You can try value_of_css_property method and with a little bit of cleanup.
The code:
# getting information from the image using `value_of_css_property method
news_image = container.find_element(by="xpath", value="./div[@class='thumbnail']").value_of_css_property('background-image')
## Output: 'url("https://cdn.animenewsnetwork.com/thumbnails/cover400x200/youtube/NKQYx6mGQPE.jpg")'
# from the get the core URL, as the value contains *url* declaration with a combination of `lstrip` and `rstrip`
news_image = news_image.lstrip('url("').rstrip('")')
## Output: https://cdn.animenewsnetwork.com/thumbnails/cover400x200/youtube/NKQYx6mGQPE.jpg
I haven't tested the code yet, but if it doesn't work, let me know :)
Update
Images on the site loads on scroll. This is the result prior to scrolling.
This after I have manually scrolled to the end:
You need to go to the bottom of the page, then wait a bit for the images to load. Here is another post relevant to this:
How to scroll down the page till bottom(end page) in the Selenium WebDriver
CodePudding user response：
Actually,image url is in
data-srcattribute value not instyleattribute. So you have to invokedata-srcinstead ofstyle.When the is oppened by the selenium then you will notice that the main data listing will not be seen. So to see you have to scroll a bit down.
For some reasons, selenium throws an error in
data-src attribute poritonwhich isno such elementBut the the same element selection works with the help of selenium bs techinique
Now, open any of the images url and see the image
Example:
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
#chrome to stay open to see what's happening in the real word or make it comment to close
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
URL ='https://www.animenewsnetwork.com/'
driver.get(URL)
scroll_to_view_elements = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[@id="mainfeed"]')))
driver.execute_script("arguments[0].scrollIntoView(true);", scroll_to_view_elements)
#containers =WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, '//div[@]')))
soup = BeautifulSoup(driver.page_source,"lxml")
containers = soup.select('.herald.box.news')
for container in containers:
#news_image = container.find_element(By.XPATH, ".//div[@class='thumbnail']").get_attribute("data-src")
news_image = 'https://www.animenewsnetwork.com' container.select_one('div.thumbnail').get("data-src")
print(news_image)
Output:
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190365/normankingstadt.png.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/NKQYx6mGQPE.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190360/visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/178241/blade-runner-key-visual-3.png.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/_MdNEjLtmeA.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190347/spy-x-family-loid-1.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/vk3Fr9ysPYo.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/186222/fb-min.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/187236/utano-princesama.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190338/the-devil-is-a-part-timer-sequel.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/186449/one-piece-film-red-visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190336/yoko-takahashi.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/183303/001_size6-15.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/183594/visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/X3GjvqjJGJ8.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.4/164788/dragon-quest-the-adventure-of-dai.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/unlvnzHlBbk.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190311/fdz7id5wiaa-s9_.jfif.jpeg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/188439/beasttamer.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/189313/visual-gundam.jpeg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/187809/hanzawa.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190319/suzume.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190309/phantom-tales-of-the-night-11.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190302/bungo-stray-dogs-22.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/185138/visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190298/maiko-irie.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190165/02_thecafe_terraceanditsgoddesses_oct22digidebut.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190299/kaguya-sama-film-visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190260/image1-1-.png.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190297/bohni_reveal_screenshot_03_wolf_3840x2160-1-.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190296/my-master-has-no-tail-mameda-father.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/YOIJOJsUkUg.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190294/fdwgshmwqaezyzp.jfif.jpeg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/VwwZU7YgC6A.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/183950/news_220324-01.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190284/popteamepic.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/188246/more-than-a-married-couple.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/daily-briefs/190251/307166287_119745634191865_3536768112624593346_n.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/txIZFXwTrtY.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190275/visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190274/visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190272/soumatezuka.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/3K9ww6eAjc4.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/herald/139673/816546026639-anime-demon-slayer-kimetsu-no-yaiba-mugen-train-arc-limited-edition-blu-ray-primary.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190264/eminence-in-shadow-game.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/188246/more-than-a-married-couple.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/HxTHBgpCMfo.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190261/yoko-nigiri-new-manga.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/185350/umayuru_teaser.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news/130178/fushigiyugi_byakko.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/185551/vermeil_visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/hxcnwH8TPEg.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190247/ragnarok.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190255/ars-kv.jpeg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190235/iwashita.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/youtube/JpzZYIi3In0.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/181939/yakuzalover.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190237/karras.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/187141/lom01.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190211/fuuto_stage_tv.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.3/157300/visual.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/189931/hypmic.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.4/174830/satoshi-kon-documentary.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/185823/goodbye-don-glees.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/190232/crescendo.jpg
https://www.animenewsnetwork.com/thumbnails/cover400x200/cms/news.5/186449/one-piece-film-red-visual.jpg