Python Selenium: How do I print the values from a website in a text file?-CodePudding

I'm trying to write a script that will grab the following 6 values from the website tulsaspca.org and print them in a .txt file.

6 Values

The final output should be:

HTML for "Animals Placed"

<span class="number" data-to="905">905</span>
</div>
<p class="title">Animals Placed</p>

I wrote the following code, but it doesn't seem to be working.

for element in driver.find_elements_by_class_name('Animals Placed'):
  print(element.text)

CodePudding user response：

I do not see HTML for all 6 numbers.

But for this HTML

<span class="number" data-to="905">905</span>
</div>
<p class="title">Animals Placed</p>

Your script should look something like this :

XPath

//p[text()='Animals Placed']/preceding-sibling::div/span[@class='number']

Please check in the dev tools (Google chrome) if we have unique entry in HTML DOM or not.

Steps to check:

Press F12 in Chrome -> go to element section -> do a CTRL F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.

Code trial 1:

time.sleep(5)
animal_num = driver.find_element_by_xpath("//p[text()='Animals Placed']/preceding-sibling::div/span[@class='number']").text
print(animal_num)

Code trial 2:

animal_num = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//p[text()='Animals Placed']/preceding-sibling::div/span[@class='number']"))).text
print(animal_num)

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Update:

Please use the below xpath

//span[@class='number' and @data-to]

it should represent all the number of nodes in HTML DOM.

driver.maximize_window()
driver.get("https://tulsaspca.org/")
driver.execute_script("window.scrollTo(0, 250)")
all_numbers = driver.find_elements(By.XPATH, "//span[@class='number' and @data-to]")
for number in all_numbers:
    print(number.text)

Output :

CodePudding user response：

To grab the six values from the website

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

PS: You may like to drop the first row from the DataFrame